The first integration with an LLM is always easy: one key, one SDK, three lines and a prompt. The second, six months later, is no longer so. A second provider appears because Claude reasons better on long tasks, or because a self-hosted model is needed for data that cannot leave the perimeter, or because someone discovers that Cohere multilingual embeddings cost a fraction of what OpenAI charges for equivalent work. At that point the application code stops being clean. Each SDK has its own client, its own message format, its own streaming semantics, its own errors, its own rules for function calling. The team starts writing adapters, and every new cross-cutting requirement \u2014 rate limiting, observability, per-tenant budget, fallback when a provider is down \u2014 has to be implemented twice or three times over.<\/p>\n
The pattern that solves this is old and familiar in infrastructure: a proxy. Instead of each application talking directly to each provider, they all talk to a single internal service that talks to the outside world on their behalf. LiteLLM<\/a><\/strong> is, as of early 2024, the most serious open-source project for doing this in the LLM space. It offers an OpenAI-compatible API over more than a hundred providers, it can be deployed as a library or as an HTTP server, and it comes with most of the things you would eventually end up writing yourself.<\/p>\n The question is not trivial, because any proxy adds latency, another component to maintain, and another failure point. The justification has to be concrete. There are four reasons, and they usually arrive together.<\/p>\n The first is homogeneity. A single OpenAI-compatible client across all applications, pointing at an internal endpoint, replaces half a dozen SDKs. Switching model becomes a configuration field, not a refactor. Migrating an entire app from GPT-4 to Claude 3 Opus is reduced to repointing an alias.<\/p>\n The second is governance. As soon as more than one team uses LLMs, someone asks how much each is spending, and ideally wants to cap it before next month\u2019s invoice surprises finance. A central proxy issues virtual keys per team, per user or per service, with budget and expiry attached. The real provider keys live in exactly one place.<\/p>\n The third is resilience. LLM providers go down, rate-limit, and serve degraded responses more often than one would expect from services at their price point. A proxy can declare fallbacks \u2014 if GPT-4 returns 429 or 5xx, retry on Claude 3 Sonnet; if Anthropic is saturated, fall back to the self-hosted Mistral \u2014 without the applications noticing. This turns provider incidents into silent degradations rather than product outages.<\/p>\n The fourth is observability. Cost, latency and token metrics by model, tenant and route, emitted from a single point to Prometheus or Langfuse, avoid having to instrument every call in every application. It is also the natural place to insert caching, PII redaction, auditing and compliance.<\/p>\n LiteLLM can be used in two modes, and the choice shapes everything else. In library mode you import In proxy mode you deploy a separate binary \u2014 container, pod, systemd unit \u2014 and the applications talk to it as if it were OpenAI. This is the default configuration for any serious use. Its cost is an internal network hop of order 5-20 ms, negligible compared with the hundreds or thousands of milliseconds of an actual LLM call. Its benefit is concentrating all cross-cutting logic in one place.<\/p>\n A typical configuration is a YAML with three blocks. The first, The minimum fragment \u2014 the only one worth the space here \u2014 captures the three pieces together:<\/p>\n The rest of the surface \u2014 per-key budgets, Redis caching with TTL, tagging by environment, Langfuse or Helicone integration, Prometheus metrics \u2014 is described in the same file with analogous blocks and applied without touching application code.<\/p>\n LiteLLM translates between different APIs, and the translation is not always perfect. The most provider-specific features \u2014 structured output with complex schemas, OpenAI function calling versus Anthropic tool use, the reasoning modes of certain models \u2014 sometimes do not map 1:1. It is worth reading the changelog before trusting a critical flow to a non-trivial translation. Added latency is small but not zero, and for high-volume embedding workloads it can be more noticeable than expected. The proxy itself is yet another piece to maintain, with its own database, upgrades and metrics. And if real usage is a single provider with no plan to change, the complexity does not pay for itself: a local abstraction layer in the backend is enough.<\/p>\n The deployment I have seen stabilise in several teams is always similar. Two replicas of the proxy behind an internal service, a shared Postgres for keys and usage, a Redis for semantic cache, virtual keys per team or service with a monthly budget, fallbacks declared for the two or three critical models, a Prometheus scrape with An LLM proxy is not a revolutionary idea, it is the same indirection layer already placed between applications and databases, between applications and queues, between applications and identity. It earns its place for the same reasons: it isolates decisions that change often, it concentrates governance and observability, and it lets the application ignore the details of the provider. LiteLLM is today the most complete open-source implementation, stable enough for production and flexible enough to absorb the changes that will keep arriving in the model stack over the next quarters. With a single provider and no foreseeable second one, the component is dispensable. From the second model onwards, giving up on hand-written adapters and delegating to a proxy stops being a matter of taste and becomes basic hygiene.<\/p>\n","protected":false},"excerpt":{"rendered":" When an application talks to two or more LLM providers, sooner or later a proxy appears in between. LiteLLM offers a concrete one, and this is an honest read of what it gains and what it costs.<\/p>\n","protected":false},"author":1,"featured_media":841,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[24,22],"tags":[261,379,382,381,51,380],"class_list":["post-838","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-herramientas","category-inteligencia-artificial","tag-anthropic","tag-litellm","tag-llm-routing","tag-multi-proveedor","tag-openai","tag-proxy"],"translation":{"provider":"WPGlobus","version":"3.0.2","language":"en","enabled_languages":["es","en"],"languages":{"es":{"title":true,"content":true,"excerpt":true},"en":{"title":true,"content":true,"excerpt":true}}},"gutentor_comment":0,"yoast_head":"\nWhy Proxy at All<\/h2>\n
Library or Server<\/h2>\n
litellm.completion<\/code> inside the application code and enjoy the unified API without deploying anything new. This is reasonable for monoliths, prototypes or one-off scripts, but it loses almost all the cross-cutting benefits: every instance of the app needs the keys, every team does its own rate limiting, every service emits metrics its own way.<\/p>\nWhat the Proxy Declares<\/h2>\n
model_list<\/code>, maps logical names like gpt-4<\/code>, claude-3-sonnet<\/code> or mistral-local<\/code> to concrete provider configurations: the prefix openai\/<\/code>, anthropic\/<\/code> or ollama\/<\/code> identifies the backend, the key is read from an environment variable, and api_base<\/code> can point at an internal Ollama. The second, router_settings<\/code>, declares routing policy and fallbacks: an ordered list per logical model indicates which others to jump to when the first fails, and a global strategy such as least-busy<\/code>, lowest-cost<\/code> or lowest-latency<\/code> decides the tie-breaker when several candidates qualify. The third, general_settings<\/code>, sets the master key used by an administrator to mint virtual keys via API, points at a Postgres to persist budgets and usage, and optionally wires a Redis for caching of semantically equivalent responses.<\/p>\n<\/a>model_list<\/span>:<\/span><\/span>\n<\/a> <\/span>-<\/span> <\/span>model_name<\/span>:<\/span> gpt-4<\/span><\/span>\n<\/a> <\/span>litellm_params<\/span>:<\/span><\/span>\n<\/a> <\/span>model<\/span>:<\/span> openai\/gpt-4<\/span><\/span>\n<\/a> <\/span>api_key<\/span>:<\/span> os.environ\/OPENAI_API_KEY<\/span><\/span>\n<\/a> <\/span>-<\/span> <\/span>model_name<\/span>:<\/span> claude-3-sonnet<\/span><\/span>\n<\/a> <\/span>litellm_params<\/span>:<\/span><\/span>\n<\/a> <\/span>model<\/span>:<\/span> anthropic\/claude-3-sonnet-20240229<\/span><\/span>\n<\/a> <\/span>api_key<\/span>:<\/span> os.environ\/ANTHROPIC_API_KEY<\/span><\/span>\n<\/a><\/span>\n<\/a>router_settings<\/span>:<\/span><\/span>\n<\/a> <\/span>fallbacks<\/span>:<\/span><\/span>\n<\/a> <\/span>-<\/span> <\/span>gpt-4<\/span>:<\/span> <\/span>[<\/span>"claude-3-sonnet"<\/span>]<\/span><\/span>\n<\/a> <\/span>routing_strategy<\/span>:<\/span> least-busy<\/span><\/span>\n<\/a><\/span>\n<\/a>general_settings<\/span>:<\/span><\/span>\n<\/a> <\/span>master_key<\/span>:<\/span> os.environ\/LITELLM_MASTER_KEY<\/span><\/span>\n<\/a> <\/span>database_url<\/span>:<\/span> os.environ\/DATABASE_URL<\/span><\/span><\/code><\/pre>\n<\/div>\nWhat Not to Expect<\/h2>\n
A Pattern That Works<\/h2>\n
model<\/code>, tenant<\/code> and route<\/code> labels, and alerts on per-provider error rate. Applications see a single OpenAI-compatible endpoint and send their virtual key in a header; everything else happens inside the proxy.<\/p>\nConclusion<\/h2>\n