Open-weight models in enterprise: one year on
Updated: 2026-07-07
A little over a year ago, when many of us started putting open-weight models into real enterprise environments, the conversation was cautious. We had a decent Llama 2, a brilliant-for-size Mistral 7B, and a handful of alternatives that fell well behind the GPT-4 of the time. A year on, the situation has changed enough to deserve an honest balance, because some bets paid off, others failed, and the decision to build on open weights or on closed APIs is now playing out on different ground.
Key takeaways
-
Parity with closed models is per task, not global: classification, extraction, and well-scoped summarization are hard to tell apart blind; complex reasoning and long code still give closed APIs an edge.
-
The real production cost is serving with acceptable latencies under reasonable concurrency; training graphs are misleading.
-
Three patterns that have settled: multi-model router, RAG with open embeddings, and LoRA fine-tuning only for closed domains.
-
Three recurring failures: underestimating operational cost, always going to the biggest model, and anchoring to one specific model without abstractions.
-
Clearly pay off for: high sustained volume, regulated data, edge/offline. In other cases, the API remains the rational choice by total operational cost.
The quality jump was real, but uneven
The first thing worth acknowledging is that the distance to closed models has clearly shrunk on nearly every task that mattered:
-
Llama 3 and 3.1 405B: showed an open lab could reach GPT-4 Turbo levels on standard evaluations. In Meta’s official announcement, the instruction-tuned 405B scored 87.3 on MMLU (5-shot) against GPT-4 Turbo’s 86.5, after training on more than 15 trillion tokens across 16,000 H100 GPUs (Meta AI, July 2024[1]).
-
DeepSeek V3 and R1: brought competitive performance at notably lower training budgets; R1 with explicit chain-of-thought reasoning. Per its technical report, V3 is a mixture-of-experts model with 671 billion total parameters (37 billion activated per token) that finished pre-training in 2.788 million H800 GPU hours, roughly $5.58 million at the rental price quoted in the report itself (DeepSeek-AI, December 2024[2]).
-
Qwen 2.5: the most solid option for Asian languages and the most predictable for code. Alibaba Cloud released it in September 2024 with over 100 open models ranging from 0.5 to 72 billion parameters and support for more than 29 languages (Alibaba Cloud, 2024[3]).
-
Mistral Large 2: a European alternative with 123 billion parameters, released under the Mistral Research License with a commercial-license option for self-deployment (Mistral AI, July 2024[4]).
The important nuance is that parity is per task, not global. For classification, extraction, and well-scoped summarization, the difference between a well-served Llama 3.1 70B and a closed API is hard to perceive blind. In long-form English writing, closed models still stay a step ahead on consistency, and in real-project coding with a large context window, Claude 3.5 Sonnet held a perceptible edge over any open weights tested as of late 2024. On complex mathematical reasoning, DeepSeek R1 was the surprise. My read is that for most real enterprise work (classifying, extracting, summarizing, rephrasing, translating, answering over documents), open weights are already enough. For the top tier, it still pays to compare.
What it really costs to serve them well
This is where the year brought the most learning. The real production cost is serving with acceptable latencies under reasonable concurrency.
A Llama 3.1 70B quantized to 4 bits fits on one 80 GB H100 and gives good latencies for a single user at a time. Scaling it to tens of concurrent requests is the hard part. The systems that have worked are, in order of preference: vLLM and TGI on Nvidia, and SGLang when aggressive sequence parallelism is needed. The original PagedAttention paper behind vLLM measured a 2-to-4x throughput improvement over earlier systems such as FasterTransformer and Orca (Kwon et al., 2023[5]). For more detail on serving, see our analysis of vLLM in 2025.
Cost versus a closed API depends heavily on volume. For a team processing tens of millions of tokens a month, renting A100 or H100 GPUs and serving with vLLM comes out at roughly half the cost of OpenAI or Anthropic, with non-trivial operational expenses. For low volumes, the API still wins from the simple absence of human overhead.
The entry-level GPU that has settled as the sweet spot is the 40 or 80 GB A100. A server with two A100s serves a quantized 70B very well for internal use. For smaller models (8-13B), RTX 4090s or L40S GPUs give a lot for comparatively little money.
Where they are fitting into real architectures
The patterns I have seen settle are relatively few:
The "router" model: a thin layer in the application decides which model handles each request based on objective criteria (input length, data sensitivity, budgeted cost, required quality). A concrete example: short, low-risk requests go to a self-hosted Llama 3.1 8B, complex requests go to Claude 3.5 Sonnet over the API, and requests over regulated data go to a self-hosted Mistral Large 2 that stays within the compliance perimeter.
RAG with open embeddings: embeddings have matured too. Models like BGE-M3, jina-embeddings-v3, and Nomic’s match closed ones and serve from tens of MB of RAM. BGE-M3, for instance, covers more than 100 languages and an 8,192-token context window while combining dense, lexical, and multi-vector retrieval in a single model (BAAI/bge-m3 model card on Hugging Face[6]). For many cases the expensive piece of a RAG has stopped being the LLM and become the indexing pipeline. For the GraphRAG side of this, see Microsoft’s GraphRAG in enterprise.
Fine-tuning for specific domains: a year of experience has convinced me that serious fine-tuning only pays off when you have a closed domain with its own vocabulary and thousands of curated examples. In every other case, prompt engineering with RAG performs better. Fine-tuning is tempting in meetings and disappointing in production if nobody is dedicated to maintaining the training pipeline. For the technical how-to, see RLHF and DPO evaluation.
Failures worth remembering
Several teams launched local-model initiatives and ended up going back to APIs. The failure patterns repeat:
-
Underestimating operational cost. A self-hosted model needs real care: monitoring, software updates, GPU management, capacity reservation, resilience against node failures. Teams without prior GPU-operations experience get caught off guard.
-
Always going to the biggest model. The 405B is impressive, but serving it costs more, and for most enterprise cases it offers no perceptible improvement over a well-quantized 70B.
-
Rigidity against evolution. Models improve every two or three months, and anchoring to one specific model means falling behind. The teams that came through the year well built abstractions to swap the model underneath and treat it as an interchangeable component.
When they pay off
Open weights clearly pay off in three situations:
-
High, sustained volume where per-token cost dominates total cost.
-
Regulated or sensitive data where leaving the perimeter is friction or simply impossible.
-
Edge/offline: private mobile apps, industrial deployments, connectivity-free scenarios.
In every other case (low volume, non-sensitive data, non-critical latency, teams without operational AI experience), APIs remain the rational choice. Not for quality, which is already equivalent, but for total operational cost.
The near future runs through smaller, specialized models. For teams deciding today, investing in an abstraction layer and an evaluation suite of your own is worth more than the concrete choice of which model to use this quarter. Models change; the engineering that wraps them, not so much.
This piece is also available in Spanish: Modelos de pesos abiertos en empresa: un año después.
