What FreeToken is and how it runs 753B MoE models on one machine

A 753-billion-parameter model fits on a desktop computer. Not because the GPU has memory to spare, but because FreeToken stops treating the machine as a small GPU and starts treating it as a system with three memory pools and two processors. This article explains the mechanism, walks through the benchmark numbers alongside the hardware that produced them, and spells out the four requirements the press coverage skips. The Spanish version is at /freetoken-motor-ia-local/.

Key takeaways

• FreeToken is an Apache 2.0 inference engine for Mixture-of-Experts models, published on 17 August 2026 by a UC Berkeley and UT Austin team that includes Matei Zaharia and Ion Stoica.
• Its central idea is a formula, q* ≈ m · BP/BH, that decides at each step how many missing experts to fetch over PCIe and how many to compute on the CPU, based on your machine’s measured bandwidth.
• On an RTX 5090 it serves Qwen3.6-35B-A3B at 77-83 tokens per second, between 1.8x and 2.3x faster than llama.cpp; on an RTX PRO 6000 it runs the 753B GLM-5.2 at 14.9 tokens per second.
• The requirement almost nobody mentions is not VRAM but system RAM: DeepSeek-V4-Flash’s expert pool takes roughly 140 GB at FP4 and lives in host memory.
• The command line requires Linux x86_64 with an NVIDIA GPU and CUDA 13. No AMD, no Apple Silicon, no Windows outside the desktop app.
• It sits at version 0.1.2, with six weeks of public repository and 247 open issues. This is early software, however promising.

What FreeToken is and what it solves

FreeToken is a model-serving engine built to run on a user’s own machine rather than in a datacenter. It specialises in one family: Mixture-of-Experts models, where each token activates only a fraction of the total parameters. GLM-5.2 has 753B parameters but activates around 40B per token; DeepSeek-V4-Flash has 284B and activates 13B.

That sparsity is exactly what makes the arithmetic viable on a modest machine, and exactly what complicates it. Per-token compute is small, but the full expert pool has to be available somewhere, because the router can request any of them at any moment. A 284B model in MXFP4 is roughly 140 GB of weights that fit in no consumer GPU.

Earlier tools solved this by freezing a strategy at load time. llama.cpp splits layers between GPU and CPU using a fixed number. KTransformers places the most frequent experts on the GPU and leaves the rest outside. Both decisions are made once, without knowing what the model will ask for next.

FreeToken’s stance is different, and its authors put it this way in the paper abstract: the system "treats a personal machine not as a small GPU, but as a unified, elastic inference platform". Instead of fixing the split, it recomputes it continuously against the resources actually present.

How the adaptive CPU-GPU split works

When the router selects a layer’s experts, some are already in the GPU cache and some are not. The paper calls the missing ones m. The operational question is what to do with them, and there are two possible paths: fetch them over the PCIe bus to the GPU, or leave them where they are and compute that part on the CPU.

FreeToken does not pick one path; it uses both at once, in the proportion that suits the machine. The policy reduces to a single expression:

q* ≈ m · BP/BH

  m   = experts missing from the GPU cache
  BP  = pinned expert-transfer bandwidth over PCIe
  BH  = host-side processing bandwidth (CPU + RAM)
  q*  = how many of those m are fetched over PCIe; the remaining m - q* run on CPU

The elegance is that the formula adapts itself to very different hardware without a code change. On the paper’s RTX 5090 the ratio between the two bandwidths is 52.7 to 77.3; on the RTX 4060 laptop it is 11.8 to 47.5, because the laptop’s PCIe link is only x8. The same piece of maths produces two very different behaviours, which is precisely the goal.

Three more pieces surround that policy:

• Global LRU expert cache. GPU residency follows what the router requests. Adjacent tokens tend to select overlapping experts, so temporal locality exists and can be exploited. The measured result: with 37% of the expert pool on the GPU, FreeToken misses 16% of the time, against 41% for KTransformers and 62% for llama.cpp.
• Double buffering during prefill. While the GPU computes layer l‘s experts from one buffer, a dedicated transfer stream loads layer l+1 into the other. The transfer hides behind the computation.
• Semantic cache anchors. In agent workflows the context is edited constantly (tool calls, thinking blocks). These anchors avoid recomputing the whole context on each edit, and that is where the difference shows most: FreeToken loses under 12% of its decode rate moving from single-turn to multi-turn, while KTransformers loses 31%.

All that dynamic control lives inside statically captured CUDA graphs, so no synchronisation is paid per step.

The benchmark numbers, with their hardware

Loose figures mean nothing without the machine that produced them. This is the evaluation table in full:

Against KTransformers the Qwen3.6 advantage is 1.5x. Time to first token stays under 44 seconds across every evaluated workload, while the reference systems exceed 150 seconds in the worst case. The paper’s baselines are llama.cpp, KTransformers, Ollama and MoE-Infinity.

That 44-second figure deserves a calm reading. It is an enormous improvement on the alternative, and it is still a very long wait compared to any cloud API. FreeToken changes what your machine can serve; it does not turn it into a datacenter.

Installing it and starting a server

The Windows and Linux desktop app is at flashml.ai and configures the engine for you. From the command line the path is this:

# Requirements: Linux x86_64, NVIDIA GPU, driver r580+ (CUDA 13), Python 3.10 or newer
uv venv && source .venv/bin/activate
uv pip install "freetoken[accel]"

## Calibrate this machine's CPU/PCIe split. Once only, and it is what
## enables the hybrid backend the paper's numbers come from.
ft bench bw

## Start the server. --model takes a path or a Hugging Face repo id.
ft serve --model Qwen/Qwen3.6-35B-A3B

## Ready when the log says: API server is ready to serve on 127.0.0.1:1919
curl http://127.0.0.1:1919/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.6-35B-A3B","messages":[{"role":"user","content":"hello"}]}'

CUDA kernels are JIT-compiled on first use, so you need CUDA 13’s nvcc on the PATH. The ft bench bw step is the one most often skipped: auto mode resolves to offload for MoE models and only upgrades to hybrid when it finds a cached bandwidth profile. Without that command, you are not measuring the engine the paper describes.

The server exposes both the OpenAI API (/v1/chat/completions, /v1/responses, /v1/models) and the Anthropic one (/v1/messages), which means any client for either works by pointing its base URL at port 1919. There is even a shortcut for coding agents:

# Writes the agent's provider config and starts it against your server
ft launch claude    # also: codex, dsh, hermes, openclaw, opencode
ft launch claude --dry-run   # preview the changes without touching anything

Supported models include DeepSeek-V4, GLM-5.2 and GLM-4.7, the Qwen3.6 and Qwen3.8 families, gpt-oss at 20B and 120B, Gemma-4, MiniMax-M2.5 and Muse-Glimmer, in MXFP4, NVFP4, FP8 and BF16. If you are coming from quantisation with llama.cpp, the shift in thinking is that Hugging Face safetensors load directly here, and GGUF is natively accepted only for Gemma-4.

The four limits almost nobody mentions

This is where the enthusiastic coverage falls short. None of these points invalidates the project, but all four change who can use it today.

1. The hidden cost is system RAM, not VRAM. The headline says "284B on a gaming PC". The small print is that DeepSeek-V4-Flash’s full expert pool takes roughly 140 GB at FP4 and sits in host memory. A well-equipped gaming machine has 32 GB. Serving that model needs a board that accepts 192 GB of DDR5, which puts the bill much closer to a workstation. Qwen3.6-35B-A3B, by contrast, does fit comfortably on a normal machine, and that is the realistic use case for most people.

2. Hardware support is narrow. NVIDIA GPUs from the RTX 30, 40 and 50 series. The install documentation asks for Linux x86_64 and driver r580 or later. There is no AMD or Apple Silicon support, so anyone working on a Mac stays with Ollama or LM Studio. Windows enters only through the desktop app.

3. It is six-week-old software. The repository was created on 20 July 2026, the only tagged release is v0.1.2 from 19 August, and there are 247 open issues against 10,488 stars. That ratio between attention and maturity warrants caution before building anything stable on top.

4. There are documented per-model traps. Qwen3.8-Flash-Next keeps a 47.7 GiB PLE table pinned in host RAM, on top of the weights. DeepSeek-V4 checkpoints must keep the inference/config.json subdirectory or the engine cannot find the authoritative model arguments. And multimodal checkpoints are served text-only.

When it beats Ollama or llama.cpp

The honest comparison depends on which model you want to run, not on which engine is "better".

FreeToken’s niche is specific and until now it was empty: open-weight MoE models too large for the GPU, on a machine with an NVIDIA card and plenty of RAM. Outside that, the familiar tools remain the right answer. If the underlying interest is the viability of open-weight models in the enterprise, this engine widens considerably what you can test without renting GPUs by the hour.

It is worth noting that the project itself acknowledges reusing design and code from SGLang, vLLM, FlashInfer, LightLLM and llama.cpp. It does not appear from nowhere: it is a careful synthesis of a decade of prior model-serving work, applied to a scenario previously treated as marginal.

Frequently asked questions

Does FreeToken work on a Mac with Apple Silicon?

No. The install documentation requires Linux x86_64 with an NVIDIA GPU and CUDA 13, and the desktop app ships for Windows and Linux only. The whole design revolves around splitting work between GPU memory and host RAM across PCIe, a problem Apple’s unified memory does not pose in the same way. On a Mac, Ollama or LM Studio remain the option.

How much RAM do I actually need?

It depends entirely on the model, and this is the figure worth calculating before downloading anything. The expert pool lives in system memory, so you need to cover the quantised model size. Qwen3.6-35B-A3B is comfortable on a 32 GB machine. DeepSeek-V4-Flash, at 140 GB in FP4, demands a workstation. VRAM determines your speed; RAM determines whether the model starts at all.

Can I use it with Claude Code or another coding agent?

Yes, and it is one of the uses the project treats as a priority. The server exposes the Anthropic API at /v1/messages alongside the OpenAI one, and the ft launch claude command writes the agent’s provider config and starts it pointing at your server. It also covers codex, hermes, openclaw and opencode. If this area interests you, the article on function calling with Ollama covers the basics of tool use with open models.

Conclusion

FreeToken changes the question. For three years, running large models on your own machine meant hunting for the smallest model that would do the job. With an adaptive split across GPU, CPU and RAM, the question becomes how much system memory you have, and that is a far cheaper constraint to lift than VRAM.

That said, version 0.1.2 and 247 open issues call for exploratory rather than production use. The sensible way to start is with Qwen3.6-35B-A3B on an NVIDIA GPU running Linux, executing ft bench bw before your first measurement so that hybrid mode comes into play. If it works on your hardware, you will have doubled the model size your machine can serve without buying anything.

Sources

1. FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution (arXiv 2608.16157)
2. Official repository and documentation on GitHub
3. freetoken 0.1.2 package on PyPI
4. Meet FreeToken, an Edge-Native MoE Serving Engine (MarkTechPost)
5. Qwen3.6-35B-A3B checkpoint on Hugging Face