vLLM on an ARM CPU or llama.cpp, which serves a small model better without a GPU

vLLM 0.30 now runs on an arm64 CPU without a GPU through its official vllm/vllm-openai-cpu image, but only with BF16, FP16 or FP32 weights; llama.cpp 0.5.0 serves the same model quantized as a GGUF. I had both servers answer with Qwen3-1.7B over the OpenAI-compatible API, in Docker, on 28 September 2026. The host is an 18-core arm64 CPU with no GPU. llama.cpp in Q8_0 generated 72.7 tokens/s on a single request, 7.2 times what vLLM managed in BF16, and with 8 concurrent requests it stayed 3.5 times ahead. Below are the commands, the errors along the way and the measurements. There is also a Spanish version of this comparison.

Key takeaways

• The vllm/vllm-openai-cpu:v0.30.0 image has an arm64 variant and takes 5.19 GB on disk (1.32 GB compressed); llama.cpp:server-v0.5.0 takes 1.2 GB (295 MB compressed).
• On a single request, llama.cpp in Q8_0 generated 72.7 tokens/s and vLLM in BF16 10.1; with 8 at once, 236.9 against 68.3 tokens/s in total.
• vLLM scaled almost linearly from 1 to 8 requests (9.9 to 68.3 tokens/s) and used 11.4 GiB of RAM; llama.cpp used 7.0 GiB.
• On Apple silicon, vLLM dies while warming up the model: GCC 15 compiles with -march=native and aborts with "no support for ‘sme’ without ‘sve2’". --enforce-eager or a compiler wrapper gets around it.
• On Arm, vLLM accepts no quantized weights, and the CPU image lacks the GGUF plugin. The model weighs 4.06 GB in BF16, against 1.83 GB for Qwen’s official Q8_0 GGUF.
• At temperature 0, vLLM in BF16 and llama.cpp in Q8_0 and in BF16 returned the same text, with 27 input tokens and 9 output tokens in all three cases.
• vLLM puts Qwen3’s reasoning inside content unless you pass --reasoning-parser qwen3; llama.cpp splits it into reasoning_content by default.

What vLLM offers on an arm64 CPU

vLLM on CPU is a basic support mode, not the project’s main path. The vLLM CPU installation guide^[1] puts it this way: "vLLM offers basic model inferencing and serving on Arm CPU platform, with support for NEON, data types FP32, FP16 and BF16". There are +cpu Python wheels for aarch64 and Docker images tagged -arm64.

Native BF16 requires ARMv8.6-A or later (the FEAT_BF16 extension), and the guide names AWS Graviton3 and Graviton4 and AmpereOne. The same page limits CPU quantization to AWQ, GPTQ and INT8 W8A8, all three on x86 and s390x only. On Arm, then, vLLM loads 16-bit weights, twice the memory of a Q8_0.

The list of models validated on CPU^[2] includes Qwen/Qwen3-1.7B, but the validated hardware is Intel Xeon 6 and Xeon 5. Nothing on that page certifies performance on Arm.

Which versions I used and on what machine

I tested the latest stable releases available on 28 September 2026:

• vLLM 0.30.0: released on 22 September 2026, with 762 commits from 315 contributors according to its release notes^[3]. For CPU it moves the Docker base to Ubuntu 25.04 with GCC 15 and adds a faster LM head on Arm, "about 6% on Llama-3.1-8B W8A8"
• llama.cpp 0.5.0: released on 23 September 2026, with ggml 0.25.0, according to its release page^[4]. The server reports build b11146-7fe450e19
• Qwen3-1.7B: Apache-2.0 licence, 1.7 billion parameters and 28 layers, according to the model card on Hugging Face^[5]. For llama.cpp I used Qwen’s official Q8_0 GGUF^[6] and a BF16 GGUF I converted myself

The host is a Linux arm64 virtual machine on Apple silicon, with 18 cores, 47 GB of RAM and no GPU. Its CPU advertises bf16, i8mm, sme and sme2, but neither SVE nor SVE2, and that detail breaks vLLM. During setup I shared it with other jobs, so I pinned each container to 7 cores with --cpuset-cpus. The measurements came later, on the machine with no other jobs, with both servers on the same 9 cores.

How to start vLLM on an arm64 CPU with Docker

vLLM starts with one docker run that mounts the model read-only and sets the KV cache and the cores. I downloaded the files of Qwen3-1.7B revision 70d244cc with curl into a shared directory ($MODELS) and started the container like this:

docker run -d --name b6p6-vllm-run -p 127.0.0.1:24601:8000 \
  --cpuset-cpus 10-16 --shm-size 4g \
  --cap-add SYS_NICE --security-opt seccomp=unconfined \
  -e VLLM_CPU_KVCACHE_SPACE=4 -e VLLM_CPU_OMP_THREADS_BIND=10-15 \
  -e CXX=/usr/local/bin/gxx-nosme \
  -v "$PWD/gxx-nosme.sh:/usr/local/bin/gxx-nosme:ro" \
  -v "$MODELS/Qwen--Qwen3-1.7B:/models/Qwen3-1.7B:ro" \
  vllm/vllm-openai-cpu:v0.30.0 /models/Qwen3-1.7B \
  --served-model-name qwen3-1.7b --dtype bfloat16 --max-model-len 4096

Each option maps to something in the CPU guide:

• VLLM_CPU_KVCACHE_SPACE: gigabytes reserved for the KV cache. It defaults to 0; with 4 the log reports "CPU KV cache size: 37,376 tokens" and a maximum concurrency of 9.12 requests of 4,096 tokens
• VLLM_CPU_OMP_THREADS_BIND: the cores for the model’s OpenMP threads. I leave core 16 free for the HTTP server, because the guide recommends reserving "1-2 CPU cores for the serving framework"
• --cap-add SYS_NICE and seccomp=unconfined: without them you get get_mempolicy: Operation not permitted, which according to the guide breaks nothing but disables per-NUMA-node memory placement
• CXX: the compiler wrapper explained in the next section. You do not need it outside Apple silicon

Why vLLM does not start on Apple silicon

Without the wrapper, vLLM loads the weights and dies while warming up the model. vLLM compiles the graph with torch.compile, and Inductor, its CPU compiler, generates C++ and builds it with -march=native. On this CPU, GCC 15 turns on SME, Arm’s matrix extension, without SVE2, and refuses to go on:

cc1plus: sorry, unimplemented: no support for 'sme' without 'sve2'
torch._inductor.exc.InductorError: CppCompileError: C++ compile error
(EngineCore pid=58) ERROR [core.py:1366] EngineCore failed to start.

The catch is that the container stays Up: the API process does not exit, but it never opens the port, and curl against /health fails with exit code 56. If you do not read the log, it looks like a slow start.

There are two ways out. The first is --enforce-eager, which according to the log itself equals "-cc.mode=none -cc.cudagraph_mode=none": it disables torch.compile and the server starts.

The second way keeps compilation and changes only the offending flag. -march=native+nosme does not work, because GCC answers "unknown value ‘native+nosme’ for ‘-march’". So I wrote a g++ wrapper and passed it in the CXX variable, which Inductor honours:

#!/bin/sh
# gxx-nosme.sh: GCC 15 enables SME without SVE2 with -march=native
for a in "$@"; do
  shift
  if [ "$a" = "-march=native" ]; then
    a="-march=armv8.6-a+bf16+i8mm+dotprod+fp16"
  fi
  set -- "$@" "$a"
done
exec /usr/bin/g++-15 "$@"

With the wrapper, the log shows "saved AOT compiled function" and the server answers. The extension list comes from this machine’s /proc/cpuinfo, so adjust it for another CPU. The failure hits CPUs with SME and without SVE2, like these Apple cores; a Graviton with SVE does not trigger it.

How to serve the same model with llama.cpp

llama.cpp serves the official GGUF from a single container with no extra privileges. This is the command I used, with 8 slots of 4,096 tokens to match vLLM’s per-request context:

docker run -d --name b6p6-llama-run -p 127.0.0.1:24602:8080 \
  --cpuset-cpus 10-16 \
  -v "$MODELS/Qwen--Qwen3-1.7B-GGUF:/models:ro" \
  ghcr.io/ggml-org/llama.cpp:server-v0.5.0 \
  -m /models/Qwen3-1.7B-Q8_0.gguf --alias qwen3-1.7b \
  --host 0.0.0.0 --port 8080 -c 32768 -np 8 -t 6

The llama.cpp server README^[7] documents that continuous batching is on by default and that -np sets the number of slots. With -c 32768 -np 8 the log confirms "n_slots = 8, n_ctx_slot = 4096". If this is your first llama.cpp install, the guide to installing llama.cpp on Linux, macOS and Docker covers the other routes.

To compare at the same data type as vLLM, I also converted the safetensors to a BF16 GGUF with the full-v0.5.0 image. The result has 311 tensors and 4.06 GB:

docker run --rm --cpus 4 \
  -v "$MODELS/Qwen--Qwen3-1.7B:/in:ro" -v "$PWD/gguf:/out" \
  ghcr.io/ggml-org/llama.cpp:full-v0.5.0 --convert --outtype bf16 \
  --outfile /out/Qwen3-1.7B-BF16.gguf /in

At startup, llama-server 0.5.0 warns about two things: "no API key is set and CORS allows all origins", and that the default port will become 9931 in a future release. Always publish the port on 127.0.0.1, or set --api-key if you expose it.

Both servers return the same text

At temperature 0 with reasoning mode off, vLLM and llama.cpp answered the same question with exactly the same text. This is the request, identical for both ports:

curl -s http://127.0.0.1:24601/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3-1.7b", "temperature": 0, "max_tokens": 60,
       "messages": [{"role": "user",
         "content": "¿Cuál es la capital de Portugal? Responde en una frase."}],
       "chat_template_kwargs": {"enable_thinking": false}}'

vLLM in BF16, llama.cpp in Q8_0 and llama.cpp in BF16 all answered "La capital de Portugal es Lisbona.", with 27 input tokens and 9 output tokens. Both servers apply the same chat template and the same tokenizer.

"Lisbona" is the Italian spelling: the mistake belongs to the 1.7-billion-parameter model, not to the server. Asked for three Spanish rivers and their seas, it sent the Tagus, the Douro and the Ebro into the "North Sea". A model this small is for classifying or summarising text you give it, not a source of facts.

The visible difference is in reasoning. Qwen3 thinks by default, and without chat_template_kwargs vLLM returns the <think> block inside content. With --reasoning-parser qwen3 it moves it to the reasoning field.

llama.cpp splits it out with no extra options, but into a field with a different name, reasoning_content. If your client reads one of the two, switching servers breaks that read.

Which weight formats each one accepts on arm64

The weight format sets the memory you need before anything else. This table sums up what I checked and what the sources say:

I tried to load the Q8_0 GGUF in vLLM and it failed with "It looks like the config file at ‘/models/gguf/Qwen3-1.7B-Q8_0.gguf’ is not a valid JSON file". The vLLM GGUF documentation^[8] explains why: support "has migrated to OOT vllm-gguf-plugin", and the CPU image does not ship that plugin. The same page calls it "highly experimental and under-optimized".

How I compared the performance of both

The comparison measures what an API client sees, not what each engine reports in its logs. I wrote a dependency-free Python script that opens streaming requests against /v1/chat/completions and records for each one:

• TTFT: time to first token, from sending the request to the first text chunk
• Per-request tokens/s: tokens generated after the first one, divided by the time since that first token
• Aggregate tokens/s: all tokens in a wave divided by the time the whole wave takes
• Memory: the container’s cgroup memory and the RSS of its processes, before and after each wave

Every request set max_tokens to 256 with ignore_eos, and every one returned exactly 256 tokens, so both servers generated the same amount. The waves sent 1, 4 and 8 requests at once, with prompts of 46 to 64 tokens; each configuration had one warm-up wave and 5 measured ones, and I report medians. Every prompt starts with a unique request number so that no prefix cache skips prompt processing.

Both servers ran on cores 0 to 8. vLLM used 8 OpenMP threads and left the ninth core to its HTTP server; llama.cpp used 6 threads. With llama-bench on those cores, 6 threads gave 80.98 tokens/s and 8 threads 47.26 with a spread of ±26.95, so the split favours vLLM, not llama.cpp. Each configuration started with a load average below 3; during the waves the servers themselves pushed it up to 11.3.

How each one performs with 1, 4 and 8 requests

llama.cpp in Q8_0 won every measurement, and the lead narrowed as concurrency grew. On one request it generated 7.2 times more tokens per second than vLLM; with 8, 3.5 times more. These are the medians:

vLLM behaved as its scheduler promises: each request held about 9.5 tokens/s even with 8 in flight, and the total grew 6.9 times. llama.cpp split its speed across the slots: each request dropped from 72.7 to 32.4 tokens/s and the total grew only 3.3 times. It started so far ahead that with 8 requests it still led by a wide margin.

With a long prompt of 2,034 tokens and 64 output tokens, the first token took 14.2 s on vLLM and 6.6 s on llama.cpp. Compiling with torch.compile bought nothing measurable: with --enforce-eager vLLM gave 10.6 tokens/s against 10.1, and its engine started in 0.19 s instead of 36.8 s.

The llama.cpp BF16 row explains much of the gap. With the same weights as vLLM, llama.cpp stayed at 4.6 tokens/s, less than half of vLLM. On this CPU the prebuilt image loads the libggml-cpu-armv8.2_2.so variant and warns that BF16 tensors "cannot be used with preferred buffer type CPU_REPACK". The weight format mattered more than the server: on Arm, quantizing to Q8_0 is the one thing vLLM cannot do.

When to pick vLLM and when to pick llama.cpp on a CPU

On an arm64 CPU, llama.cpp was faster at 1, 4 and 8 requests, used 4.4 GiB less RAM and starts with no extra privileges or workarounds. vLLM needed twice the memory for the weights and, on Apple silicon, a workaround to start. In return it offers the same API and the same server you would use with a GPU, Prometheus metrics on /metrics and a scheduler that batches concurrent requests at every step. If you already serve with vLLM and its high GPU throughput in production, the CPU variant lets you test without switching tools.

Pick llama.cpp to serve a small model on an arm64 CPU: with 8 concurrent requests it delivered 3.5 times vLLM’s total. Consider vLLM only if you already run it on GPUs and want the same tool for testing, or if you need each request to keep its speed as users grow. Measure with your own model first, because in this test it did not catch llama.cpp even at 8 requests. To pick the model itself, the comparison of CPU-only LLMs for an arm64 machine measures three candidates with llama.cpp.

Frequently asked questions

Does vLLM work on a Mac with Apple silicon?

The CPU guide describes native macOS support as experimental: you build from source and it only handles FP32 and FP16. The 0.30.0 notes already list CPU wheels for macOS. Inside a Linux arm64 virtual machine, as in this test, the vllm-openai-cpu image works once you get around the SME without SVE2 error, with --enforce-eager or with the g++ wrapper.

Can I use a GGUF model with vLLM on a CPU?

Not with the official CPU image. vLLM moved GGUF support into an external plugin that the image does not include, and on Arm it does not accept AWQ, GPTQ or W8A8 either. For a quantized model on an arm64 CPU, llama.cpp is the direct option.

Do I need to turn off Qwen3’s reasoning to serve it?

No, but decide it in the client. With chat_template_kwargs and enable_thinking set to false, both servers answer without a reasoning block. If you leave it on, start vLLM with --reasoning-parser qwen3, or the <think> text ends up mixed into the answer.

Conclusion

vLLM 0.30 serves a small model on an arm64 CPU through the same API as llama.cpp 0.5.0, and at temperature 0 both returned the same text. On performance there was no tie: llama.cpp in Q8_0 generated 72.7 tokens/s on one request and 236.9 with 8, against 10.1 and 68.3 for vLLM in BF16. It also used 4.4 GiB less RAM.

vLLM scales well with concurrency, but on Arm it cannot quantize, and that weighs more. To repeat the test on your CPU, use the commands above and compare Q8_0 against BF16 before comparing server against server.

Sources: [1] vLLM CPU installation guide^[1], [2] vLLM models validated on CPU^[2], [3] vLLM 0.30.0 release notes^[3], [4] vllm/vllm-openai-cpu tags on Docker Hub^[9], [5] vLLM GGUF documentation^[8], [6] llama.cpp 0.5.0 release^[4], [7] llama.cpp server README^[7], [8] Qwen3-1.7B model card on Hugging Face^[5], [9] Official Qwen3-1.7B GGUF^[6].

Sources

1. vLLM CPU installation guide
2. models validated on CPU
3. release notes
4. release page
5. model card on Hugging Face
6. Qwen’s official Q8_0 GGUF
7. llama.cpp server README
8. vLLM GGUF documentation
9. vllm/vllm-openai-cpu tags on Docker Hub