The pattern is predictable. A team deploys its first production agent, the first month’s invoice arrives, and it’s double or triple the estimate. The default reaction is panic followed by aggressive optimisation, sometimes at cost to quality. There’s a third way: disciplined FinOps applied to agents, with levers ordered by cost/risk ratio.

Key takeaways

  • Prompt caching cuts input token cost 50-90% with zero quality impact.

  • A model router saves 30-70% by assigning each task to the right model.

  • Context control prevents the quadratic per-turn cost growth.

  • Batching for non-interactive tasks offers 50% discounts from Anthropic and OpenAI.

  • Without per-tenant and per-task telemetry, none of the other levers can be calibrated.

Lever one: aggressive caching

The easiest win with zero quality impact is caching. Claude prompt caching[1], OpenAI prompt caching[2] and equivalents reduce input token cost by 50-90% when there’s structural repetition. Most applicable pattern: long system prompts with instructions, few-shots, or stable tool context between calls.

Typical implementation: everything that doesn’t change between turns is marked cacheable; what changes (user query, dynamic state) stays outside. Savings are immediate and require no changes to agent logic.

Lever two: model router by difficulty

Not every task needs the most expensive model. A router classifying the query and routing to the right model saves 30-70% depending on task distribution. Classification can be:

  • Simple: keyword rules.

  • Sophisticated: small model as classifier (Haiku 4.5 as router).

The common stack combines Haiku 4.5 or Gemini Flash for light tasks, Sonnet 4.6 for most traffic, and Opus 4.7 only for queries the router flags as complex. This idea of choosing a model based on the request is the core of the savings. Key: measure router accuracy. If it misclassifies and sends the expensive model when unnecessary, savings evaporate.

Lever three: context control

Agents tend to accumulate context. Unchecked, a five-turn conversation reaches twelve thousand tokens; ten turns, twenty-five thousand. Per-turn cost grows quadratically because accumulation is billed on every call.

Techniques that work:

  • Periodic summarisation: every N turns, history compresses to a summary.

  • Sliding window: only the last K full turns.

  • Retrieval selection: at each turn’s start, recover only relevant fragments.

Combined, they cut per-conversation spend by 40-60% without user-perceived quality loss if well calibrated.

Lever four: batching where possible

Non-interactive tasks (nightly processing, report summarisation, bulk classification) accept batching. Anthropic Batch API[3] and OpenAI Batch API[4] offer 50% discounts for tolerable latency (hours instead of seconds). For flows where immediate response isn’t required, not batching is leaving money on the table.

Lever five: telemetry revealing real spend

Necessary condition for optimising is seeing where money goes. Minimum is metrics by:

  • Tenant and task.

  • Model and call type (cacheable or not).

  • Input and output tokens.

At that granularity, tenants consuming 10× the mean, poorly scoped tasks, and broken flows making looped calls surface. Tools including this telemetry: Helicone[5], Langfuse[6], Portkey[7], plus native provider dashboards. I cover what to instrument first in LLM observability: traces, costs, and quality.

What doesn’t work

Three frequent antipatterns:

  • Switching provider for 10% price differences without changing anything else: engineering time exceeds savings.

  • Downgrading model without evaluating: quality drops, customers complain, rollback with net loss.

  • "Negotiating with provider" without real volume: volume discounts start where the top 1% of customers by spend sit; below that threshold there’s no room to negotiate.

Conclusion

Agent FinOps is a mature area with clear levers. Applied in order (caching, routing, context control, batching, and telemetry) they cut cost by half or a third without perceived quality impact. I go into more depth on this instrument-first approach in FinOps applied to AI: where the cost really goes. What doesn’t work is reacting to the invoice with panic and cutting visible things; what works is investing a few days in instrumentation and architectural decisions that pay back in month one.

Frequently asked questions

How much does prompt caching save, and does it hurt quality?

Prompt caching (Claude prompt caching, OpenAI prompt caching and equivalents) reduces input token cost by 50-90% when there is structural repetition, with zero quality impact. It applies best to long system prompts, few-shots, or stable tool context between calls: everything that does not change between turns is marked cacheable and what changes stays outside. Savings are immediate and require no changes to the agent logic.

Can I downgrade the model to save money without losing quality?

Downgrading the model without evaluating is an antipattern: quality drops, customers complain, and you roll back with a net loss. What works is a router that classifies each query and sends it to the right model, saving 30-70%. The split: Haiku 4.5 or Gemini Flash for light tasks, Sonnet 4.6 for most traffic, and Opus 4.7 only for queries flagged as complex. The key is measuring the router's accuracy.

Why does an agent's per-turn cost grow so much in long conversations?

Because agents accumulate context and that accumulation is billed on every call, so per-turn cost grows quadratically: unchecked, a five-turn conversation reaches twelve thousand tokens and ten turns twenty-five thousand. The techniques that contain it are periodic summarisation every N turns, a sliding window keeping only the last K full turns, and retrieval selection of relevant fragments. Combined, they cut per-conversation spend by 40-60%.

Sources

  1. Claude prompt caching
  2. OpenAI prompt caching
  3. Anthropic Batch API
  4. OpenAI Batch API
  5. Helicone
  6. Langfuse
  7. Portkey