Benchmarks and the memory ceiling in oMLX
Table of contents
- Key takeaways
- The panel's benchmark tool
- What it measures exactly
- How to measure without fooling yourself
- The community leaderboard
- The memory ceiling, which decides before anything else
- The errors you see when it does not fit
- What the cache changes in the numbers
- Frequently asked questions
- Why do my oMLX benchmarks come out worse the first time?
- How much context fits on my Mac?
- Can I raise the memory ceiling above the default?
- Conclusion
- Sources
A tokens-per-second figure that does not say which context, which batch size and whether the cache was warm or cold means nothing. oMLX ships a built-in benchmark tool that measures the four variables that do matter, and a memory ceiling that decides before anything else whether the test even fits. This article covers both, and the error messages you see when it does not.
Key takeaways
- The benchmark tool lives in the admin dashboard and measures throughput and quality separately.
- There are four metrics: time to first token, time per output token, generation tokens per second and peak memory.
- Prompts run from 1,024 to 200,000 tokens, and batches are tested at 2x, 4x and 8x.
- The default memory ceiling is system RAM minus 8 GB, adjustable through
memory_guard_tieracross four levels. - Without warmup mode enabled, the first measurement includes just-in-time compilation and comes out consistently worse.
The panel’s benchmark tool
It is built into the admin dashboard and has two modes answering different questions.
The throughput and latency mode measures speed: how long the model takes to start answering, at what rate it generates, and how much memory it peaks at. It is the one you use to decide whether a model is usable on your machine.
The accuracy evaluation mode runs standardised benchmarks such as MMLU, GSM8K and HumanEval. It answers a different question: whether the model, with the quantisation you gave it, still reasons well. This part matters a lot when you drop bit widths, because the quality cost is not linear.
Results reach the browser over Server-Sent Events with a replay-on-subscribe model, which means you can close the tab, come back and still watch the run. Small detail, but a 200,000-token context run takes a while.
What it measures exactly
Four metrics, and it is worth not confusing them because they measure different phases of the same request:
| Metric | What it is | What makes it worse |
|---|---|---|
| TTFT | Time from sending the request to the first token | A long prompt, a cold cache, first-use compilation |
| TPOT | Time per token during generation | Model size, quantisation, memory contention |
| Tokens per second | Generation throughput | The same as TPOT, expressed the other way round |
| Peak memory | Maximum in use, read from the engine itself | Context, batch size, model size |
The split between TTFT and TPOT is the one most often ignored and the one that explains the most. Prefill, which is processing the whole prompt, and generation, which is producing token by token, scale differently: the first grows with context length, the second barely does. A model can take fifteen seconds to get going on a 100,000-token context and then generate at a good rate. If you only watch tokens per second, you will not see that problem.
The 0.6.4 release notes illustrate it well: the improvement measured on an Apple M3 Ultra was 33.5% in prompt processing and 24.2% in total request time at 32K context. Two different figures for the same change, because they touch different phases.
How to measure without fooling yourself
Three benchmark parameters change the result more than the model does:
Warmup mode. It forces just-in-time compilation before measurement starts. Without it, the first pass carries that cost and comes out worse than the ones after. Always turn it on, unless what you want to measure is precisely how long a cold start takes.
Prompt length. The menu runs from 1,024 to 200,000 tokens. Measure at the length you will actually use. A figure obtained at 1,024 tokens says nothing about how the model behaves with a whole code file.
The cache test. There are two scenarios, same prompt and different prompt, comparing a warm prefix cache against a cold one. The gap between them is exactly what the tiered cache saves you on your real workload. If you always work over the same long context, the same-prompt scenario is your case and the different-prompt one is pessimistic.
Batch size (2x, 4x and 8x) only matters if you are going to serve concurrent requests. For a single person in front of an assistant, a batch of one is the honest number.
The community leaderboard
Accuracy results can be uploaded to omlx.ai, where they feed a community leaderboard. The upload goes in two steps: first the hardware metadata and final scores, then the gzipped raw per-question results.
It is useful for comparing your machine against others on the same chip, with the usual caveat: these are user-submitted measurements, on configurations you do not control.
The memory ceiling, which decides before anything else
A Mac with unified memory has no separate video memory. Every loaded model comes out of the same budget as the operating system and every other application, so oMLX imposes a limit of its own to stop the whole Mac running out.
That limit defaults to system RAM minus 8 GB. On a 128 GB Mac that leaves 120 GB for models and cache; on a 16 GB one it leaves 8 GB, which is little for almost anything useful.
The level is set through memory_guard_tier, which takes four values: safe, balanced (the default), aggressive and custom. With custom you set the ceiling yourself in memory_guard_custom_ceiling_gb. From the command line these are --memory-guard and --memory-guard-gb.
There is also a prefill guard, prefill_memory_guard, on by default, which checks whether the request fits before it starts processing it. That is what turns an out-of-memory crash into a readable error.
{
"memory_guard_tier": "custom",
"memory_guard_custom_ceiling_gb": 96,
"prefill_memory_guard": true,
"max_concurrent_requests": 8,
"hot_cache_max_size": "20%",
"ssd_cache_max_size": "auto"
}
Watch out for hot_cache_max_size: it ships as "0", which disables the hot cache in RAM. If you want hot blocks staying in memory you have to give it a value, as a percentage or in gigabytes. ssd_cache_max_size ships as "auto", resolving to 10% of the disk capacity.
The errors you see when it does not fit
The messages people run into come in two families, and they mean different things.
The ones about the prefill not fitting in available memory come from the prefill guard. The prompt you sent, with that model and that context, needs more memory than is left under the ceiling. It is not that the model does not fit: it is that the prompt does not. The ways out are shortening the context, unloading another model or raising the ceiling.
The ones about something not fitting under the memory ceiling come from the general guard, while trying to load a model. Here the problem is the whole model, and the ways out are a more aggressive quantisation, pinning fewer models or changing tier.
Before touching the ceiling, look at what is loaded. Least-recently-used eviction frees models on its own, but a pinned model is never unloaded and a long TTL keeps models you are not using alive. The sharing is covered in detail in model management and memory in oMLX.
Raising the ceiling above the default is tempting and usually goes badly. That 8 GB reserve is not free-floating conservatism: it is what macOS needs to avoid starting to compress memory, at which point performance falls much further than the extra context gains you.
What the cache changes in the numbers
The tiered key-value cache alters TTFT dramatically and does not touch TPOT. Hot blocks live in RAM and cold ones spill to SSD in safetensors format; when a request arrives with the same prefix, they are restored from disk instead of recomputed, and that works even after a server restart.
On top of that, TurboQuant quantises the cache itself with a mean-squared-error codec and a Lloyd-Max quantiser, and cuts its size by 60% to 75%. Less memory per context token means more context under the same ceiling.
The practical consequence when measuring is that you have to decide what you want to know. With a warm cache you measure the good case, a conversation that continues. With a cold cache you measure the bad case, the first request of the day. Both numbers are true and they describe different moments.
Frequently asked questions
Why do my oMLX benchmarks come out worse the first time?
Because the first pass includes just-in-time compilation and, if the prefix cache is cold, recomputing the whole prompt as well. Turn on the benchmark tool’s warmup mode and discard the first measurement.
How much context fits on my Mac?
It depends on the memory ceiling, which defaults to system RAM minus 8 GB, on how much the loaded model takes, and on how far TurboQuant compresses the cache. Rather than calculating it, measure it: run the benchmark raising the prompt length until the prefill guard rejects the request.
Can I raise the memory ceiling above the default?
Yes, with memory_guard_tier set to custom and a value in memory_guard_custom_ceiling_gb. Do it carefully: the 8 GB reserve exists so macOS does not start compressing memory, and when that happens you lose more performance than the extra context wins.
Conclusion
Measuring well in oMLX is mostly about deciding what you are measuring. Separate prefill from generation, always state which context and which cache state, and turn warmup on. With that, the panel’s benchmark figures are comparable across models and across machines; without it, they are noise.
The memory ceiling is the other side of the same coin, because it sets the upper bound on everything else. If you have just installed the server, configuration is in the API and port guide, and the benchmark tool is found in the admin dashboard. The Spanish version of this article is at Benchmarks y techo de memoria en oMLX.
Sources
Source code
Access all the source code for this post on GitHub.
View on GitHub
