vLLM in 2025: the improvements that matter to LLM-serving teams
Updated: 2026-07-07
Two years ago, serving language models in production was an exercise in fragmentation. Every team hitting the problem ended up choosing among a dozen options, and the decision was rarely final. Today, while serious alternatives persist, vLLM has become the default engine for most teams serving models on GPU. And the growth isn’t accidental: it’s the result of a very consistent improvement pace over two years.
This post reviews important vLLM changes over the last months and frames them in terms of what they mean for operators.
Key takeaways
-
Automatic prefix caching is the most impactful improvement: for requests that repeat a long prefix, time-to-first-token can drop from several seconds to under a second, without changing anything in the application.
-
Speculative decoding reduces latency especially for 70B+ models, but adds operational complexity.
-
Multi-LoRA support transforms the economics of multi-tenant services: one shared base model + per-customer adapters.
-
Multi-GPU support remains more fragile than single-GPU for some large models.
-
Non-NVIDIA hardware (AMD ROCm, Intel Habana) lags in experience and maturity.
The maturity moment
vLLM started as an academic project focused on PagedAttention (efficient KV memory management during inference) and has grown into a platform with a predictable release cycle, stable API, and enterprise user ecosystem. Consolidation shows in details: better documentation, more honest published benchmarks, first-class integration with orchestration frameworks (Ray, Kubernetes).
Most relevant technically: the original PagedAttention paper[1] already documented a 2-4x throughput improvement over comparable systems of the time (FasterTransformer, Orca) at the same latency, by cutting KV-cache memory waste from a typical 60-80% down to under 4%. That architectural advantage has held up through later improvements, and it translates directly into infrastructure cost.
Prefix caching: what’s changed most
The most impactful improvement is automatic prefix caching. When many requests share a prefix (typical case: an application’s system prompt, or a RAG’s common context), vLLM detects the overlap and reuses the already-computed attention cache. In a test published by the llm-d project[2], a request with a roughly 10,000-token prompt against a Qwen3-32B instance went from 4.3 seconds to first token down to just 0.6 seconds the second time, purely from prefix caching. And aggregate throughput scales proportionally.
Integration is frictionless: nothing to configure, works automatically. For a team already using vLLM, moving to the prefix-caching version is just an upgrade.
Speculative decoding: the second big improvement
The technique uses a small fast model to predict several tokens ahead and then verifies with the main model. If predictions are correct, the big model validates in a single pass what would have required several, and effective latency drops.
vLLM has incorporated speculative decoding with several draft-model options (EAGLE, MTP, n-gram, dedicated draft models), per the official documentation[3]. Latency improvement is especially noticeable in large models (70B+). For interactive workloads where user experience depends on time-to-first-token, it’s a qualitative change.
The consideration is that speculative decoding adds operational complexity: you need to deploy the draft model alongside the main one.
Multi-LoRA: a specific case
For teams serving multiple fine-tunes of the same base model (typical in multi-tenant SaaS where each customer has their adapter), vLLM has matured multi-LoRA support significantly. Per the LoRA Adapters documentation[4], you can load a base model plus several concurrent LoRA adapters (the limit is set by the max_loras parameter), and each request specifies which adapter to use as if it were a separate model, without reloading the base model.
This transforms the economics of multi-tenant LLM services. Instead of deploying a model per customer, you deploy a shared base model and an adapter per customer. Adapters are small (a few MB each), so many can sit active in GPU memory at once, far beyond what deploying a full model per customer would allow.
Comparison with alternatives
Serious alternatives remain Hugging Face’s TGI and NVIDIA’s TensorRT-LLM.
TGI has improved a lot and now has features comparable to vLLM in most areas. Good option if already integrated in the Hugging Face ecosystem. It also has a concrete strength: per a technical comparison published on MarkTechPost[5], on very long prompts (over 200,000 tokens) TGI v3 can serve a reply in about 2 seconds versus 27.5 seconds for vLLM, thanks to prefill chunking. For workloads dominated by long context and reuse (RAG over large documents), it is worth evaluating.
TensorRT-LLM offers the highest throughput on NVIDIA hardware when you can dedicate time to specific optimization (the same comparison cites figures above 10,000 output tokens per second on H100 for well-compiled workloads). The price is a more complex compilation pipeline. For high-volume services with predictable workloads, worth considering; for services with variable workloads or frequent model changes, vLLM remains more comfortable.
llama.cpp and derivatives (Ollama, LM Studio) don’t compete on throughput but on simplicity. Excellent for prototypes and local applications; for services handling tens or hundreds of concurrent requests, vLLM is superior by design.
What remains a weak point
-
Multi-GPU support has improved but remains more fragile than single-GPU.
-
Non-NVIDIA hardware support lags. vLLM works on AMD with ROCm, but the experience is clearly inferior to NVIDIA.
-
Memory consumption during startup is high. vLLM loads the model and KV cache buffers aggressively. For large models on GPUs with limited VRAM, fitting can be hard.
What it means for operators
For practically any team serving LLMs on NVIDIA GPU with non-trivial workloads, vLLM is the option with the best return on time investment. Recent improvements (prefix caching, speculative decoding, multi-LoRA) have widened the lead over alternatives.
My recommendation to a team starting today:
-
Start on the latest stable version.
-
Measure with your real workload before micro-optimizing.
-
Enable prefix caching from the start if your prompts have repeated parts.
-
Consider speculative decoding only if measurements show latency is a real issue.
-
Don’t try to tune all knobs at once.
Medium-term, I expect vLLM to maintain its improvement pace and become taken-for-granted infrastructure, the way Redis or PostgreSQL are in their respective niches today. For people building products on LLMs, that stability is good news: less time on infrastructure, more on the product.
