Updated: 2026-07-07

Testing systems that contain language models breaks the fundamental axiom on which the entire automated-testing discipline was built: given the same input, the system produces the same output. Generative models, by their own nature, do not guarantee that property even at zero temperature: Anthropic’s own Messages API documentation[1] warns that, even with temperature at 0.0, "the results will not be fully deterministic," and OpenAI acknowledges the same limitation with its seed parameter, built for outputs that are "mostly" reproducible, not guaranteed[2]. There is variation between versions, between providers, between hardware, and sometimes simply between successive runs. After more than a year integrating language models into products with real users, this article collects a set of strategies that work and another set that doesn’t.

Key takeaways

  • Traditional testing with exact assertions breaks with language models: the same call can return slightly different sentences without anything having changed in your code.

  • The three-layer strategy works: deterministic code around the model (classic unit tests), tolerant snapshots by semantic similarity, and LLM-as-judge evaluations for critical cases.

  • The offline evaluation belt doesn’t measure absolute quality but drift: what catches problems is the score suddenly dropping, not a specific number.

  • Evaluation cases must be adversarial, not representative: a belt that replicates average usage gives high, stable scores that catch nothing.

  • No test belt replaces watching real users use the product.

Why classical tests break

Traditional testing rests on exact assertions: the result of this function must be this string, this number, this structure. With a language model, the same call can return slightly different sentences, a different order of items, a different emphasis in the argument. If you write an exact assertion against the output, the test will fail the next day without anything having changed in your code. The natural reaction is to lower the bar: check only that the response contains a keyword, or that it exceeds a minimum length. The problem is that those assertions are so loose they let serious regressions through undetected.

The second wall is that models evolve underneath you. If your product depends on an external provider, the model you call today is not strictly the same one you’ll call three months from now, even if the name doesn’t change. Providers like OpenAI and Anthropic update models regularly. A test belt that worked perfectly can start failing without you having touched the code. In classical engineering that would be unacceptable; in AI systems it’s the normal state.

The third problem is external dependency. Every test run consumes tokens, costs money, has network latency, and can fail on quota limits or provider outages. A 500-test belt can take 15 minutes and cost several euros per run, which makes it unfeasible to launch on every push to continuous integration. This forces you to segregate fast deterministic tests from slow stochastic ones, with a different operational discipline for each.

The three-layer strategy

The solution that has worked, and that is crystallizing across the industry, is a testing architecture with three clearly differentiated layers, each with its own goals and costs.

Layer 1: the code around the model

The first layer is the code around the model, which must be fully deterministic and tested with classical assertions. All the logic of prompt composition, response parsing, error handling, format normalization, and deciding when to call the model belongs here. This layer can and should have high coverage of fast unit tests that never touch the real model, using recorded or fixed responses. In the projects analyzed, this layer concentrates 70 to 80 percent of the tests and runs in under ten seconds.

Layer 2: tolerant snapshots

The second layer is tolerant snapshot testing. The model’s real output is captured in a reference run, saved as a snapshot, and later runs are compared against it not word for word but with a semantic similarity metric. If the current output resembles the snapshot by more than 90 percent on sentence embeddings, it passes. If it drifts far, the test fails and a human decides whether the difference is an improvement or a regression.

Layer 3: automated-judge evaluation

The third layer is automated-judge evaluation, popularly called LLM-as-judge. A model different from the one that produced the answer evaluates whether the answer meets specific criteria: is it coherent with the context, does it answer the user’s question, does it contain no detectable hallucinations, does it keep the requested tone. The technique was formalized in 2023 in the paper «Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena»[3], where automated judges like GPT-4 matched human preferences in more than 80 percent of cases, a level of agreement comparable to the one between two human evaluators. This is the most powerful layer but also the most expensive and the slowest; we cover it in more depth in when to trust an LLM-as-judge. It’s usually reserved for a curated set of critical cases and run before important releases.

Offline evaluations: what actually catches regressions

The piece that ties the three layers together is the offline evaluation belt. It’s built from a set of use cases representative of your product, each with its input, an expected output or a set of acceptance criteria, and it runs every time you change the prompt, the model, or the temperature. Open tools like the OpenAI evals framework[4] formalize this same idea as a reusable registry of test cases, with the same logic we cover when detailing the framework for production-grade agent evaluations. Results aggregate into a global score compared against the previous run.

What matters is that the belt doesn’t measure absolute quality but drift. A product can work perfectly even if its belt scores 78 percent; what catches problems is that the score suddenly drops to 65. That’s why it pays to:

  • Invest in diverse cases covering the scenarios where you know your system can fail: ambiguous inputs, sensitive content, secondary languages, edge-case formats.

  • Version the belt as code: every change to the cases should be justified and reviewed like any other change.

A finding that’s hard to internalize is that evaluation cases must be adversarial, not representative. A belt that replicates average usage gives high, stable scores that catch nothing. A belt built from the cases where the product is most likely to fail, the rarest inputs, the edges of the domain, the abuse attempts, is the one that genuinely warns you when something changes.

Continuous integration: the cadence hierarchy

The operational discipline that has ended up being most useful distinguishes three testing cadences by frequency and cost:

  • Per-push cycle: runs only the first layer, fast and deterministic, on every push to integration. Takes seconds and runs on every change.

  • Nightly cycle: runs the second layer over a stable subset of cases, at bounded cost, and flags drift the next day if there is any.

  • Weekly cycle: runs the full third layer, with automated judging, over the entire belt; this is the one that catches the subtlest regressions and the one that justifies running before an important release.

This split has cultural consequences. Accepting that not every regression is caught at the moment of the change, but in the following cycle, forces you to design deployments with easy rollback windows and with production observability that catches what the tests don’t. This connects with what we described in AI agent observability: the test belt and production monitoring are complementary layers, not substitutes for each other.

How to think about the decision

AI testing requires giving up the nostalgia for the exact assertion and accepting a probabilistic model of quality. Systems that integrate language models can’t be guaranteed with the precision of classical deterministic systems. What can be guaranteed is that there is no unacceptable drift, that critical cases keep working, and that there are monitoring and rollback mechanisms to catch what the tests miss.

The practical consequence is that teams who want to integrate AI seriously must invest proportionally more in evaluation and observability than in classical coverage. A product with a language model needs fewer traditional unit tests and more semantic evaluation tests, fewer exact assertions and more continuous metrics. It’s not less discipline; it’s different discipline.

Conclusion

No test belt replaces watching real users use the product. Belts catch known regressions; users find the unknown ones. Detecting issues in production, with conversation logs and satisfaction surveys, must be a central piece of the strategy.

This article is also available in Spanish.

Sources

  1. Messages API documentation
  2. outputs that are "mostly" reproducible, not guaranteed
  3. «Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena»
  4. OpenAI evals framework