The first integration with an LLM takes one key, one SDK, three lines and a prompt. The second, six months later, does not. A second provider appears because Claude reasons better on long tasks, or because a self-hosted model is needed for data that cannot leave the perimeter. Another reason is that Cohere multilingual embeddings cost a fraction of OpenAI’s equivalent.

At that point application code stops being clean. Each SDK has its own client, message format, streaming semantics, errors, and function-calling rules. The team starts writing adapters, and every cross-cutting requirement (rate limiting, observability, per-tenant budget, fallback when a provider is down) has to be implemented twice or three times over.

The pattern that solves this is old and familiar in infrastructure: a proxy. Instead of each application talking directly to each provider, they all talk to a single internal service. LiteLLM[1] is, as of early 2024, the most serious open-source project for doing this in the LLM space. It offers an OpenAI-compatible API over more than a hundred providers, deploys as a library or HTTP server, and comes with most of what you would eventually write yourself.

Key Takeaways

  • An LLM proxy is not a revolutionary idea. It is the same indirection layer already placed between applications and databases, queues, and identity.

  • The four reasons that justify a proxy are: API homogeneity, governance (virtual keys + budgets), resilience (automatic fallbacks), and centralised observability.

  • Server mode is what matters for serious use: library mode loses almost all cross-cutting benefits.

  • Translation between APIs is not always perfect: the most provider-specific features (structured output, tool use) sometimes do not map 1:1.

  • With a single provider and no plan to change, the complexity does not pay; a local abstraction is enough.

Why Proxy at All

The justification must be concrete, because any proxy adds latency and another component to maintain. There are four reasons, and they tend to arrive together.

The first is homogeneity. A single OpenAI-compatible client across all applications, pointing at an internal endpoint, replaces half a dozen SDKs. Switching models becomes a configuration field, not a refactor.

The second is governance. Once more than one team uses LLMs, someone asks how much each is spending, and wants to cap it. A central proxy issues virtual keys per team, user, or service, with budget and expiry. Real provider keys live in exactly one place.

The third is resilience. LLM providers fail, rate-limit, and degrade responses at a rate teams do not plan for. A proxy can declare fallbacks (if GPT-4 returns 429 or 5xx, it retries with Claude 3 Sonnet) without the applications noticing. This turns provider incidents into silent degradations rather than product outages.

The fourth is observability. Cost, latency, and token metrics by model, tenant, and route, emitted from one point to Prometheus or Langfuse, avoid instrumenting every call in every application. Also the natural place for caching, PII redaction, auditing, and compliance.

Library or Server

LiteLLM has two modes. Library mode: import litellm.completion inside application code, no new deployment needed. Reasonable for monoliths, prototypes, one-off scripts, but loses almost all cross-cutting benefits.

Proxy mode: deploy a separate binary (container, pod, systemd unit) and applications talk to it as if it were OpenAI. This is the configuration for any serious use. Cost is an internal network hop of 5-20 ms, negligible versus the hundreds or thousands of milliseconds of an actual LLM call.

What the Proxy Declares

Minimal working configuration:

model_list:
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-3-sonnet
    litellm_params:
      model: anthropic/claude-3-sonnet-20240229
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  fallbacks:
    - gpt-4: ["claude-3-sonnet"]
  routing_strategy: least-busy

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

model_list maps logical names to provider configurations. router_settings declares routing policy and fallbacks. general_settings sets the master key, points at Postgres for persisting budgets and usage, and optionally wires Redis for semantic cache. The rest of the surface (per-key budgets, caching with TTL, Langfuse or Helicone integration, Prometheus metrics) is described in the same file without touching application code.

What Not to Expect

LiteLLM translates between different APIs, and translation is not always perfect. Provider-specific features (structured output with complex schemas, OpenAI function calling vs Anthropic tool use, reasoning modes) sometimes do not map 1:1. Added latency is small but not zero, and can be more noticeable than expected for high-volume embedding workloads.

The proxy itself is another piece to maintain. And if real usage is a single provider with no plan to change, the complexity does not pay: a local abstraction layer in the backend is enough.

A pattern that works

The deployment that stabilises in teams with serious use is always similar. Two proxy replicas behind an internal service, shared Postgres for keys and usage, Redis for semantic cache, virtual keys per team or service with monthly budget. On top of that come fallbacks declared for the critical models, Prometheus scrape with model, tenant, and route labels, and alerts on per-provider error rate. Applications see a single OpenAI-compatible endpoint and send their virtual key in a header; everything else happens inside the proxy.

Conclusion

An LLM proxy is not a revolutionary idea. It is the same indirection layer already placed between applications and databases, queues, and identity. It earns its place for the same reasons: isolates the decisions that change most, concentrates governance and observability, and lets the application ignore provider details.

LiteLLM is the most complete open-source implementation, stable enough for production and flexible enough to absorb the changes that will keep arriving in the model stack. With a single provider and no foreseeable second, the component is dispensable. From the second model onwards, giving up on hand-written adapters and delegating to a proxy stops being a matter of taste and becomes basic hygiene.

Frequently asked questions

Should I run LiteLLM as a library or as a proxy server?

Proxy mode for any serious use. Library mode (importing litellm.completion in your code) needs no deployment and is fine for monoliths, prototypes and one-off scripts. But it loses almost all the cross-cutting benefits: virtual keys with budgets, fallbacks, central observability. The server is a separate binary applications talk to as if it were OpenAI, and its cost is an internal hop of 5-20 ms, negligible against an LLM call.

Does LiteLLM map every provider feature 1:1?

No. It translates between different APIs, and provider-specific features such as structured output with complex schemas, OpenAI function calling versus Anthropic tool use, or reasoning modes sometimes do not map cleanly. Added latency is small but not zero and can show up on high-volume embedding workloads, and the proxy is another component to maintain. With a single provider and no plan to change, a local abstraction layer in the backend is enough.

How do I make a provider outage not break my application?

Declare fallbacks in router_settings: for example fallbacks: - gpt-4: ["claude-3-sonnet"], so a 429 or 5xx from GPT-4 is retried against Claude 3 Sonnet without the applications noticing. Provider incidents become silent degradations instead of product outages. The pattern that stabilises in production adds two proxy replicas behind an internal service, shared Postgres for keys and usage, Redis for semantic cache and alerts on per-provider error rate.

Sources

  1. LiteLLM