Artificial Intelligence
vLLM on an ARM CPU or llama.cpp, which serves a small model better without a GPU
On an arm64 CPU with no GPU, llama.cpp 0.5.0 served Qwen3-1.7B as a Q8_0 GGUF at 72.7 tokens/s per request and 236.9 tokens/s with 8 concurrent requests. vLLM 0.30 in BF16 reached 10.1 and 68.3 tokens/s and used 11.4 GiB of RAM against 7.0 GiB.
