Tag
oMLX ships a benchmark tool in the panel that measures time to first token, time per output token, tokens per second and peak memory, with prompts from 1,024 to 200,000 tokens. The default memory ceiling is system RAM minus 8 GB, and that is what decides which context fits.
oMLX is a local inference server for Apple Silicon Macs that wraps Apple's MLX framework in a web process and exposes it through the OpenAI and Anthropic APIs. It adds continuous batching, on-disk KV caching and several models in memory at once, all driven from the menu bar.
The oMLX panel lives at localhost:8000/admin and gathers monitoring, model loading and unloading, chat with history, a Hugging Face downloader, benchmarks and per-model settings. The command line covers the rest with four commands: serve, start, stop and launch, the last one for wiring up external clients.
oMLX listens on 127.0.0.1:8000 by default and exposes eight endpoints compatible with OpenAI and Anthropic. The API key travels in the Authorization header or in x-api-key, supports sub-keys that only call the API, and can be skipped for local connections. Everything is configurable through settings.json, environment variables or flags.
oMLX installs through Homebrew in two commands: first the tap for jundot's repository, then the formula. You end up with a launchd service that starts via omlx start, a log inside Homebrew's var directory and an upgrade path through brew upgrade. The current version is 0.6.4, released on 29 August 2026.
oMLX manages each model's lifecycle with four pieces: a downloader that pulls weights from Hugging Face, aliases that rename the model in the API, a TTL that unloads it after idle time, and a tiered KV cache that spills blocks to SSD when RAM fills up.
oMLX MCP is a bridge that exposes your local oMLX server as Model Context Protocol tools. The mcp_omlx package connects Claude Desktop to the models running on your Mac and gives you seven tools to list models, load and unload them from memory, and run inference without leaving the client.
Recipe for oMLX on a Mac M5 Max with 128 GB: install, TurboQuant at 3.5-bit, the Qwen 3.6 35B-A3B model stack, Claude Code wiring and our own benchmarks. Revised for oMLX 0.6.4, with what changed since 0.3.8.
