Prompt caching reuses the stable part of your prompt (instructions, documents, tool definitions) so the model doesn’t have to reprocess it on every call, which makes requests cheaper and lowers latency. When you build an agent that repeats the same context over and over, that repetition dominates the bill. In this guide you will see what prompt caching is, how it works on Anthropic, OpenAI and Gemini, how to order the prompt to maximise hits, how much you really save and how to cache on your own machine with vLLM. The same explanation is available in Spanish.

Key takeaways

  • Prompt caching stores the already-computed prefix of a prompt; on later requests that start the same way, the model reads the cache instead of reprocessing those tokens.
  • Anthropic bills cache reads at around 10% of the input price (up to 90% cheaper) and cache writes at 1.25× (5-minute cache) or 2× (1-hour cache).
  • OpenAI applies a 50% discount to cached tokens automatically, from 1,024 tokens and in 128-token increments, with no code changes.
  • Google Gemini discounts cached tokens by 90% (2.5 Flash drops from 0.30 to 0.03 dollars per million) and enables implicit caching by default on 2.5 and newer models.
  • The cache is an exact prefix: a single-byte change at the start of the prompt invalidates everything after it, so ordering matters more than the cache markers.

What is prompt caching?

Prompt caching is a technique that stores the intermediate result of processing the start of a prompt so that work isn’t repeated on later requests. A language model turns every input token into an internal representation before generating the answer; that computation costs time and money. If ten calls share the same 8,000 words of instructions and documentation, without caching you pay to process those 8,000 words ten times.

With caching on, the first call computes and stores the prefix, and later ones read it at a fraction of the price. The saving is largest exactly where it hurts most: agents with a long system prompt, assistants answering questions about the same document, or context engineering flows that drag a lot of history along. The key is that caching works by prefix: it reuses the common tokens from the start and stops paying off at the first point where two prompts differ.

How does it work on Anthropic, OpenAI and Gemini?

All three major providers offer prompt caching, but with different activation models and pricing.

Anthropic uses explicit caching: you mark up to four breakpoints with cache_control and decide which prefix to cache. The official documentation sums up the economics: "cache reads cost only 10% of the base input price." Writes carry a small premium (1.25× for the 5-minute cache, 2× for the 1-hour one), so it pays off from the second read onward. The render order is tools → system → messages, and the minimum cacheable prefix is around 1,024 tokens depending on the model.

OpenAI uses automatic caching: you touch nothing. Any request over 1,024 tokens reuses the previously seen prefix, cached in 128-token chunks, with a 50% discount on the reused portion and up to 80% lower latency. Because writes are free on the classic models, any hit is money saved with no break-even calculation.

Google Gemini blends the two ideas. Implicit caching is on by default on 2.5 and newer models and applies the discount with no configuration. Explicit caching lets you create reusable cache objects in exchange for an hourly storage cost (1.00 dollar per million tokens per hour on 2.5 Flash). In both cases the discount on the cached token is 90%.

How to structure the prompt to maximise hits?

There is one golden rule: stable content goes first, changing content goes last. Because the cache is an exact prefix, a single different character at the start throws away all the reuse after it. This example with the Anthropic SDK caches a long document in the system block and leaves the variable question outside the breakpoint:

from anthropic import Anthropic

client = Anthropic()

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_DOCUMENT,  # the stable prefix you want to cache
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "Summarise the key points"}],
)

# Check the hit: if it is 0 on repeated requests, something is invalidating it
print(response.usage.cache_read_input_tokens)

Avoid the silent invalidators: don’t put datetime.now() or random IDs in the system prompt, serialise JSON with sorted keys, and don’t change the tool set mid-conversation (tools render first, and changing them invalidates the whole cache). For a one-hour cache, use {"type": "ephemeral", "ttl": "1h"}, which survives longer between traffic bursts at the cost of a pricier write.

How much do you save in cost and latency?

It depends on the hit ratio, but the order of magnitude is clear. If 80% of your input tokens are a shared prefix and that prefix is read at 10% of its price, the input cost drops sharply: you pay full price only for the 20% that varies plus a fraction for the rest. With OpenAI’s 50% the saving is smaller but arrives free and effortlessly.

The break-even point matters when there’s a write premium. With Anthropic’s 5-minute cache, two requests already pay off (1.25× write plus 0.1× read versus 2× uncached); with the 1-hour cache you need at least three. Latency also falls: OpenAI cites up to 80% less prompt-processing time, because reading from cache is far faster than recomputing. In an agent that fires many calls per task, that cut shows up in both the bill and the experience. If you want to measure the impact in production, a tool like Helicone shows you the hit ratio per request.

Can you cache on models you run yourself (vLLM)?

Yes, and with no per-token cost. Inference engines like vLLM implement automatic prefix caching (APC): they store the key-value cache (KV cache) of already-computed prefixes and reuse it when another request starts the same way. It isn’t a fee you save but GPU compute you don’t repeat, which translates into lower latency and more requests per second.

# Prefix caching is on by default in vLLM's V1 engine;
# this flag makes it explicit
vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-prefix-caching

It’s the same idea as the providers’ caching, applied to your own hardware: if you serve a fixed system prompt to thousands of users, the first request warms the KV cache and the rest reuse it. It pairs well with a gateway like OpenRouter when you mix your own models and API ones behind the same interface.

Frequently asked questions

Does prompt caching change the model’s answer?

No. The cache only avoids recomputing the prefix; the model produces exactly the same output it would without it. Caching is a cost and latency optimisation, not a change of behaviour. The only thing that varies is the token breakdown in the response: you will see part of the input billed as a cache read instead of as new input.

How long does the cache live?

Not long, by design. Anthropic uses 5 minutes by default and offers a 1-hour option; OpenAI retains the prefix for a matter of minutes of inactivity (with longer windows on recent models); Gemini keeps an explicit cache for the lifetime you set, paying hourly storage. Each access usually refreshes the timer, so steady traffic keeps the cache warm on its own.

What is the minimum size to be cached?

Around 1,024 tokens on Anthropic and OpenAI, and 2,048 on Gemini 2.5 Pro. Below that threshold the request simply doesn’t cache, with no error: you’ll see zero cache-read tokens. That’s why prompt caching shines on long, repeated prompts, not on short, one-off questions.

Conclusion

Prompt caching is one of the best effort-to-reward optimisations when you build agents or assistants that repeat context. The recipe is simple: put the stable content first, mark the breakpoint where the variable content starts and measure the hit ratio. With discounts of up to 90% on Anthropic and Gemini and 50% automatically on OpenAI, plus vLLM’s free prefix caching on your own machine, repetition stops being a drag on the bill. The next step is to review your system prompt, freeze its prefix and turn caching on with whichever provider you use.

Sources

  1. Prompt caching, Anthropic documentation
  2. Prompt caching, OpenAI guide
  3. Context caching, Gemini API
  4. Automatic prefix caching, vLLM documentation

Route: AI Agents in Production: Evaluation and Reliability