Two years ago, serving language models in production was an exercise in fragmentation. Every team hitting the problem ended up choosing among a dozen options, and the decision was rarely final. Today, while serious alternatives persist, vLLM has become the default engine for serving models on GPU. And the growth isn’t accidental: it’s the result of a consistent improvement pace over two years.

This post reviews important vLLM changes over the last months and frames them in terms of what they mean for operators.

Key takeaways

  • Automatic prefix caching is the most impactful improvement. For requests that repeat a long prefix, time-to-first-token can drop from several seconds to under a second, without changing anything in the application.

  • Speculative decoding reduces latency especially for 70B+ models, but adds operational complexity.

  • Multi-LoRA support transforms the economics of multi-tenant services: one shared base model + per-customer adapters.

  • Multi-GPU support remains more fragile than single-GPU for some large models.

  • Non-NVIDIA hardware (AMD ROCm, Intel Habana) lags in experience and maturity.

The maturity moment

vLLM started as an academic project focused on PagedAttention (efficient KV memory management during inference) and has grown into a platform with a predictable release cycle, stable API, and enterprise user ecosystem. Consolidation shows in details: better documentation, more honest published benchmarks, first-class integration with orchestration frameworks (Ray, Kubernetes).

Most relevant technically: the original PagedAttention paper[1] already documented a 2-4x throughput improvement over comparable systems of the time (FasterTransformer, Orca) at the same latency. The key is cutting KV-cache memory waste from a typical 60-80% down to under 4%. That architectural advantage has held up through later improvements, and it translates directly into infrastructure cost.

Prefix caching: what’s changed most

The most impactful improvement is automatic prefix caching. When two or more requests share a prefix (typical case: an application’s system prompt, or a RAG’s common context), vLLM detects the overlap and reuses the already-computed attention cache.

In a test published by the llm-d project[2], a request with a roughly 10,000-token prompt against a Qwen3-32B instance took 4.3 seconds to first token. The second time it took just 0.6 seconds, purely from prefix caching. And aggregate throughput scales proportionally.

Integration is frictionless: nothing to configure, works automatically. For a team already using vLLM, moving to the prefix-caching version is just an upgrade.

Speculative decoding: the second big improvement

The technique uses a small fast model to predict a block of tokens ahead and then verifies with the main model. If predictions are correct, the big model validates in a single pass what would have required one pass per token, and effective latency drops.

vLLM has incorporated speculative decoding with four draft-model options (EAGLE, MTP, n-gram, dedicated draft models), per the official documentation[3]. Latency improvement is especially noticeable in large models (70B+). For interactive workloads where user experience depends on time-to-first-token, it’s a qualitative change.

The consideration is that speculative decoding adds operational complexity: you need to deploy the draft model alongside the main one.

Multi-LoRA: a specific case

For teams serving multiple fine-tunes of the same base model (typical in multi-tenant SaaS where each customer has their adapter), the multi-LoRA support in vLLM has matured. Per the LoRA Adapters documentation[4], you can load a base model plus as many concurrent LoRA adapters as the max_loras parameter allows. Each request specifies which adapter to use as if it were a separate model, without reloading the base model.

This transforms the economics of multi-tenant LLM services. Instead of deploying a model per customer, you deploy a shared base model and an adapter per customer. Adapters are small (a few MB each), so far more of them can sit active in GPU memory at once than deploying a full model per customer would allow.

Comparison with alternatives

Serious alternatives remain Hugging Face’s TGI and NVIDIA’s TensorRT-LLM.

TGI has improved a lot and now has features comparable to vLLM in most areas. Good option if already integrated in the Hugging Face ecosystem.

It also has a concrete strength: on long prompts (over 200,000 tokens) TGI v3 can serve a reply in about 2 seconds, versus 27.5 seconds for vLLM. The reason is its prefill chunking, per a technical comparison published on MarkTechPost[5]. For workloads dominated by long context and reuse (RAG over large documents), it is worth evaluating.

TensorRT-LLM offers the highest throughput on NVIDIA hardware when you can dedicate time to specific optimization. The same comparison cites figures above 10,000 output tokens per second on H100 for well-compiled workloads. The price is a more complex compilation pipeline. For high-volume services with predictable workloads, worth considering; for services with variable workloads or frequent model changes, vLLM remains more comfortable.

llama.cpp and derivatives (Ollama, LM Studio) don’t compete on throughput but on simplicity. Excellent for prototypes and local applications; for services handling tens or hundreds of concurrent requests, vLLM is superior by design.

What remains a weak point

  • Multi-GPU support has improved but remains more fragile than single-GPU.

  • Non-NVIDIA hardware support lags. vLLM works on AMD with ROCm, but the experience is clearly inferior to NVIDIA.

  • Memory consumption during startup is high. vLLM loads the model and KV cache buffers aggressively. For large models on GPUs with limited VRAM, fitting can be hard.

What it means for operators

For practically any team serving LLMs on NVIDIA GPU with non-trivial workloads, vLLM is the option with the best return on time investment. Recent improvements (prefix caching, speculative decoding, multi-LoRA) have widened the lead over alternatives.

My recommendation to a team starting today:

  • Start on the latest stable version.

  • Measure with your real workload before micro-optimizing.

  • Enable prefix caching from the start if your prompts have repeated parts.

  • Consider speculative decoding only if measurements show latency is a real issue.

  • Don’t try to tune all knobs at once.

Medium-term, I expect vLLM to maintain its improvement pace and become taken-for-granted infrastructure, the way Redis or PostgreSQL are in their respective niches today. For people building products on LLMs, that stability is good news: less time on infrastructure, more on the product.

Frequently asked questions

Do I need to configure anything to benefit from vLLM prefix caching?

No: it works automatically with nothing to configure, so for a team already on vLLM it is just an upgrade. When two or more requests share a prefix (the application's system prompt or a RAG's common context), vLLM reuses the already-computed attention cache. In a test published by the llm-d project, a roughly 10,000-token prompt against Qwen3-32B went from 4.3 seconds to first token down to 0.6 seconds the second time.

Is speculative decoding worth enabling on my service?

Only if measurements show latency is a real issue. The improvement is especially noticeable on 70B+ models where each token costs a lot, and vLLM offers four draft-model options (EAGLE, MTP, n-gram or dedicated draft models). In exchange it adds operational complexity: you need to deploy the draft model alongside the main one. Measure with your real workload before micro-optimizing.

How do I serve a different fine-tune per customer without deploying a model for each one?

With multi-LoRA support: you load one shared base model plus as many concurrent LoRA adapters as the max_loras parameter allows. Each request specifies which adapter to use as if it were a separate model, without reloading the base model. Adapters are a few MB each, so far more of them can sit active in GPU memory at once than full models would, which transforms the economics of multi-tenant SaaS.

Sources

  1. original PagedAttention paper
  2. llm-d project
  3. official documentation
  4. LoRA Adapters documentation
  5. technical comparison published on MarkTechPost