Tag
On an arm64 CPU with no GPU, llama.cpp 0.5.0 served Qwen3-1.7B as a Q8_0 GGUF at 72.7 tokens/s per request and 236.9 tokens/s with 8 concurrent requests. vLLM 0.30 in BF16 reached 10.1 and 68.3 tokens/s and used 11.4 GiB of RAM against 7.0 GiB.
vLLM remains the reference engine for serving LLMs on GPU in 2025: automatic prefix caching sharply cuts latency for repeated prompts, speculative decoding speeds up large models, and multi-LoRA support lowers the cost of multi-tenant SaaS, though multi-GPU support and non-NVIDIA hardware remain weak points.
vLLM serves language models with PagedAttention, continuous batching and an OpenAI-compatible API. In October 2026 it is on 0.32.0, V1 has been the only engine since 0.11 and prefix caching is on by default. This guide covers starting it with vllm serve, quantization by hardware, metrics and when to pick something else.
SGLang adds a Python DSL for controlling LLM generation with constrained decoding, parallel branching, and RadixAttention, the structure that indexes the KV cache as a radix trie to reuse shared prefixes across requests. When that pattern exists, speedups over vLLM reach up to 5 times; without it, the advantage shrinks.
