vLLM: Serving LLMs in Production with Very High Throughput

Tested with vLLM 0.31.0 · 0.32.0 · CPU arm64 · Qwen3-0.6B · verified 2026-10-27

Updated: 2026-10-27

vLLM^[1] is the reference server for serving open language models when aggregate throughput matters: it manages the KV cache with PagedAttention, batches requests continuously and exposes an OpenAI-compatible API. On 27 October 2026 the latest release is 0.32.0, published on 18 October. Since 0.11.0, from October 2025, V1 has been the project’s only engine, and prefix caching has been on by default since 0.8.0. This guide gathers what you need to deploy it, what changed in 2025 and 2026 and when another tool fits better. It is also available in Spanish.

Key takeaways

• PagedAttention splits the KV cache into fixed-size blocks and cuts memory waste below 4 %, against 60 % to 80 % in earlier systems, according to the project’s announcement.
• Since 0.11.0 (2 October 2025) V1 is the only engine, and prefix caching and chunked prefill are on by default.
• The server starts with vllm serve <model> and listens on port 8000 with /v1 routes compatible with OpenAI clients.
• Quantization depends on the hardware: AWQ and GPTQ work from Turing on, FP8 W8A8 needs Ada, Hopper or an AMD GPU, and on Arm CPUs there is only INT8.
• Hugging Face’s Text Generation Inference is in maintenance mode and its repository is archived; the project itself recommends vLLM or SGLang.

Why does PagedAttention change throughput?

Earlier inference servers reserved a contiguous block of memory per request for the KV cache, sized for the maximum length. With requests of different lengths, much of that memory sat reserved and unused. PagedAttention chops the cache into small blocks and assigns them with a page table, the same way an operating system manages RAM. The UC Berkeley team described it in the PagedAttention paper^[2], which measured 2 to 4 times the throughput of FasterTransformer and Orca at the same latency.

Continuous batching builds on that: requests join and leave the batch at every step, without waiting for the previous group to finish. The original vLLM announcement^[3] put the result at up to 24 times the throughput of Hugging Face Transformers. Those figures are the project’s own, from 2023 and on GPU; the advantage depends on how many concurrent requests you receive.

What changed in vLLM in 2025 and 2026?

The underlying change is the V1 engine, which replaced the original one in four steps documented in each release’s notes:

Automatic prefix caching is the improvement with the most effect on real applications. When two or more requests share the start of the prompt, such as an assistant’s system message or a RAG’s common context, vLLM reuses the KV blocks it already computed. The llm-d project published a test^[4] with a prompt of about 10,000 tokens against Qwen3-32B: time to first token dropped from 4.3 s to 0.6 s the second time.

The other advanced features have matured too:

• Speculative decoding: a draft method proposes tokens and the main model verifies them. The documentation^[5] offers EAGLE, MTP, n-grams, suffix decoding and draft models, and validates that greedy sampling gives the same result with and without it.
• Multi-LoRA: one base model with more than one LoRA adapter loaded at once, up to the --max-loras limit; each request picks its own by model name, according to the LoRA adapters guide^[6].
• Structured output: JSON against a schema, regular expressions or grammars with the xgrammar or guidance backends, through the structured_outputs field that replaced the guided_* ones in 0.12.0, according to the structured outputs documentation^[7].
• Hardware: besides NVIDIA, the installation guide^[8] lists AMD ROCm, Intel XPU, Apple Silicon through vLLM-Metal and x86, Arm and IBM Z CPUs, with TPU, Gaudi and others through plugins.

How to install and start vLLM

vLLM installs as a Python package or runs from its Docker image, and a single command starts the server. The quickstart guide^[9] recommends uv to install it with the right PyTorch backend for your GPU:

uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen3-8B \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 32768

The server listens on http://localhost:8000/v1 and any OpenAI API client works by changing the base URL. For Docker, the vllm/vllm-openai image uses the same vllm serve as its entrypoint, so you only pass the model and the options. These are the parameters that weigh most:

• --tensor-parallel-size: how many GPUs in the node split the model; 1 by default.
• --gpu-memory-utilization: the fraction of GPU memory vLLM reserves for weights and KV cache; 0.92 by default.
• --max-model-len: the maximum context length; if you leave it out, vLLM takes the model’s and reserves cache for it.
• --enable-prefix-caching: already on; --no-enable-prefix-caching turns it off if you need to measure without it.

There is no GPU on our test machine, so we have not run the command above. We did run the vllm/vllm-openai-cpu image on 27 October 2026 on an 18-core arm64 CPU, with Qwen3-0.6B and --enforce-eager. Release 0.32.0 started, reported enable_prefix_caching=True and enable_chunked_prefill=True, and its first request brought the engine down with RuntimeError: Boolean value of Tensor with more than one value is ambiguous. Release 0.31.0, with the same command, answered on /v1/chat/completions without errors.

With 0.31.0 we measured prefix caching with three different questions behind the same 3,918-token system message and max_tokens set to 1. The first request took 13.63 s, and the next two 1.16 s and 1.02 s:

At the end, the vllm:prefix_cache_hits_total metric showed 7,680 reused tokens out of 11,770 queried. The two repeated requests only had to compute the question’s tokens.

Quantization and parallelism by hardware

Quantization shrinks the weights’ memory and leaves more room for the KV cache, but each format only works on certain hardware. The compatibility table in the documentation^[10] sums up the most used formats like this:

On Ampere, FP8 only works through the Marlin kernels, which quantize the weights but not the activations. Weight memory is the parameter count times the bytes per parameter. A 70-billion-parameter model takes about 140 GB in BF16 and about 35 GB at 4 bits, before counting the KV cache.

Tensor parallelism splits each layer across the GPUs in a node and is the first option when the model does not fit on one. Pipeline parallelism splits the layers across nodes and adds network latency to every step, so it is kept for when one node is not enough.

What about a CPU without a GPU?

vLLM runs on a CPU, but on an arm64 CPU a server like llama.cpp with quantized weights leaves it far behind. We measured it in vLLM on an ARM CPU vs llama.cpp, with Qwen3-1.7B in Docker on 28 September 2026 and vLLM 0.30.0. llama.cpp at Q8_0 generated 72.7 tokens/s with one request, 7.2 times what vLLM gave in BF16, and with 8 concurrent requests it was still 3.5 times ahead:

vLLM scaled almost linearly from 1 to 8 requests, as its scheduler promises, but it started at 9.9 tokens/s. On a CPU, vLLM makes sense for testing the same server you will use with a GPU. For real serving without a GPU, the guide on how to install llama.cpp is a better starting point.

Observability: which metrics to watch

vLLM publishes Prometheus metrics on /metrics, on the same port as the API. These are the ones worth having on a dashboard from day one, with the names 0.31.0 exposed in our test:

• vllm:num_requests_running and vllm:num_requests_waiting: requests in progress and in the queue.
• vllm:kv_cache_usage_perc: KV cache occupancy.
• vllm:time_to_first_token_seconds: histogram of the time to first token.
• vllm:e2e_request_latency_seconds: histogram of each request’s full latency.
• vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total: their ratio is the prefix cache hit rate.

If the queue grows steadily with the KV cache near 100 %, the limit is capacity: add replicas or a GPU with more memory. To track the cost and quality of responses, connect these metrics to LLM observability with traces.

vLLM vs SGLang, TensorRT-LLM and TGI

The choice of servers changed in 2026, and the main news is the exit of Text Generation Inference. Hugging Face put it in maintenance mode, archived its repository^[11] and recommends vLLM or SGLang in its README. Its latest release is 3.3.7, from December 2025.

SGLang is the closest open alternative and stands out on workloads with a lot of shared prefix. Its 0.5.22 release is from 16 October 2026. TensorRT-LLM squeezes NVIDIA hardware in exchange for a more complex per-model build, and its 1.4 branch is at release candidate. llama.cpp and Ollama do not compete on aggregate throughput but on ease of deployment. They are the choice for a laptop, a CPU or a prototype, as the guide on how to install Ollama explains.

When does serving your own model with vLLM pay off?

Serving with vLLM pays off when you have sustained concurrent traffic, an open model that covers your case and someone who can run it. The cost comes down to three figures: GPU hours per month, tokens served in those hours measured with your load and engineering hours for operations. Compare it with the per-token price of the API you would replace; there is no universal threshold, because it depends on the model, the GPU and how much you benefit from prefix caching.

The usual mistake is treating vLLM as a binary you switch on and forget. It needs the maximum context sized, the quantization chosen for the GPU, replicas decided and the queue watched. If you also need to adapt the model, LoRA and QLoRA produce adapters that vLLM serves with multi-LoRA on a single base model, the architecture we describe in open-weight models in enterprise.

Frequently asked questions

Do I have to change my application code to move from the OpenAI API to vLLM?

No. vllm serve exposes /v1 routes compatible with the OpenAI API on port 8000, such as /v1/chat/completions and /v1/models. Any OpenAI SDK works by changing the base URL and the model name, and passing a dummy key if the server does not require authentication.

Do I need to turn on prefix caching?

Not since 0.8.0, from March 2025: it is on by default, and 0.31.0 reported enable_prefix_caching=True at startup. In our CPU test, the second request with the same 3,918-token system message took 1.16 s, against 13.63 s for the first. To turn it off, use --no-enable-prefix-caching.

What happened to the "vLLM in 2025" article?

Its content, prefix caching, speculative decoding, multi-LoRA and the comparison with other servers, is updated in this guide. The vLLM in 2025 article stays as a snapshot of that moment, when the V1 engine had just arrived.

Conclusion

vLLM is still the most practical option for serving open models on GPU with concurrent traffic. PagedAttention and continuous batching make the most of the card’s memory, the V1 engine ships with prefix caching on and the OpenAI-compatible API saves changes to the application. With TGI archived, its direct open alternative is SGLang. On a CPU it works, but llama.cpp with quantized weights performs far better, as our arm64 measurement showed.

Sources: [1] Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention^[2], [2] vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention^[3], [3] vLLM 0.11.0 release notes^[12], [4] vLLM quantization documentation^[10], [5] llm-d, KV-Cache Wins You Can See^[4], [6] Text Generation Inference repository^[11].

Sources

1. vLLM
2. PagedAttention paper
3. original vLLM announcement
4. published a test
5. documentation
6. LoRA adapters guide
7. structured outputs documentation
8. installation guide
9. quickstart guide
10. compatibility table in the documentation
11. repository
12. vLLM 0.11.0 release notes