Local LLM Calculator

How memory is calculated

The memory a model needs is the sum of its weights, its KV cache and the engine overhead. The calculator compares it with the memory the platform lets it use, not the memory it has. Everything is in GiB, the unit VRAM and RAM are sold in.

• Weights: total parameters × bits per weight / 8. A MoE loads every expert even though it uses few per token. Bits per weight are the effective figures for each format: Q4_K_M takes 4.89 bits according to llama.cpp’s table measured on Llama 3.1 8B, and a 4-bit MLX model with groups of 64 takes 4.5.
• KV cache: elements per token × bytes per element × context × concurrent requests. The elements come from each model’s config.json. Sliding-window layers keep only the window when the engine does so, latent attention (MLA) stores one compressed vector per layer, and linear or Mamba layers keep a fixed-size state.
• Engine overhead: compute buffers and backend context. It is a fixed figure per engine, noted next to the result.

With long contexts, the KV cache sets the maximum context. Gemma 3 27B, for example, has a 1,024-token window on 52 of its 62 layers. At 32,768 tokens its FP16 cache takes about 2.9 GiB; counting every layer as full attention would give 15.5 GiB.

Why the inference engine changes the answer

The same model in the same format can fit one engine and not another, because each reserves memory its own way. These are the rules the calculator applies, read from each version’s code or documentation:

When an engine does not document a behaviour, the calculator assumes the expensive case. Two details stand out when the docs are checked against the code: vLLM reserves 92% by default although its docs say 0.9, and oMLX keeps 6 GiB free although its README says 8.

Colibri is calculated separately, because it does not load the model into memory. It keeps the dense part in RAM and reads the experts the router picks from the SSD. So the calculator compares your RAM with the minimum the project publishes and your free disk with the model size.

The disk sets its speed: the cold ceiling divides the SSD read speed by the expert GB each token reads. The project documents that figure only for GLM-5.2 (12.7 GB) and DeepSeek V4.1 Flash (4.5 GB), and for OLMoE we derived it from its configuration. The mechanism is explained in what Colibri is and how it streams experts from disk.

How much memory each platform can use

On a discrete GPU the model gets all the VRAM, times the number of cards. On unified memory it does not, and the gap is large:

• Apple Silicon Mac: macOS lets the GPU use 2/3 of RAM at 32 GiB or less and 3/4 above, per the decompiled kernel code in llama.cpp discussion 2182. A 128 GB Mac offers about 96 GiB. Raise the limit with sudo sysctl iogpu.wired_limit_mb; it resets on reboot.
• AMD Ryzen AI Max+ 395: on Windows, AMD lets you dedicate up to 96 GB of 128 to the GPU. On Linux the GPU uses half of RAM by default, and the kernel limit can be raised.
• NVIDIA DGX Spark: there is no fixed split. The OS sees about 119.7 GiB of the 128 GB and the GPU uses whatever is free.
• CPU only: the calculator keeps 4 GB for the operating system. That is an assumption, not a measurement.

How generation speed is estimated

Generating text for a single request is limited by memory bandwidth: every token reads the active weights and that conversation’s KV cache. The theoretical ceiling is the bandwidth divided by those bytes, and real speed lands below it.

The real fraction was calibrated on 132 published llama-bench measurements on NVIDIA, AMD and Apple GPUs and on CPUs. It is not constant. Below a 30 tok/s ceiling engines reach 84% of it (median); between 120 and 250 tok/s, only 58%. Small models on fast memory hit compute limits first.

The calculator shows the interquartile range of the matching band and does not extrapolate past the highest ceiling measured, 523 tok/s.

MoE models have their own calibration, from 53 published measurements of gpt-oss, Qwen3, GLM-4.7-Flash, DeepSeek R1 and others. They reach 41% of their ceiling (median), against 58-84% for dense models, because every token pays for routing and attention even though it reads few weights. There are no MoE measurements with MLX, so Macs use the same calibration measured with llama.cpp.

Self-hosting vs the API

Self-hosting has a high fixed cost, the amortized hardware, and a low marginal cost, the electricity. The API is the opposite: nothing fixed and a price per token. That is why self-hosting only pays off above a certain volume, the break-even point the tool computes.

Throughput, power draw and price follow the chosen platform until you type your own value. Prices are the official ones where the vendor publishes them. They are NVIDIA’s recommended price in its Spanish store, Apple’s Spanish store for each memory size or, failing those, the US launch price.

What it does not calculate

The calculator does not estimate prompt processing speed or batched throughput, and it does not split a model between GPU and RAM when it does not fit. Those cases depend on hardware and on each engine’s settings more than on a formula. Model, platform and engine data were verified on 15 September 2026.

Sources:

1. llama.cpp v0.4.1 bits-per-weight table
2. llama.cpp measurements on Apple Silicon
3. llama.cpp measurements with Vulkan
4. gpt-oss measurements with llama.cpp
5. GPU memory limit on macOS
6. Context length in Ollama v0.34.0
7. Quantized KV cache in vLLM v0.29.0
8. Quantized KV cache in SGLang v0.5.19
9. oMLX v0.6.4 memory ceiling
10. EXL3 conversion in ExLlamaV3 v1.5.0
11. AMD Variable Graphics Memory
12. NVIDIA DGX Spark hardware
13. Colibri v1.11.0 per-model requirements