Updated: 2026-07-07

Building a RAG system is relatively straightforward. Deciding whether it works well is much harder. Many teams deploy systems that look impressive in internal demos and disappoint in production because they never established a rigorous evaluation process. This article covers how to measure RAG quality honestly, what metrics to use, how to build evaluation sets, and the most common mistakes.

Key takeaways

  • Intuition deceives: a handful of engineer-built questions is not rigorous evaluation.

  • The four key dimensions are faithfulness, answer relevance, context precision, and context coverage.

  • The golden set is the most valuable asset: 100-500 curated questions representing real, difficult, and out-of-scope cases.

  • LLM as judge is scalable but has biases: periodically validating with humans is mandatory.

  • Integrating evaluation into the CI pipeline is the difference between improving and believing you’re improving.

Why intuition deceives

The natural temptation when evaluating a RAG system is to try a handful of questions, see seemingly reasonable answers, and declare success. This approach fails for several reasons:

  • The questions an engineer constructs to test are different from those real users will ask.

  • Easy cases are over-represented.

  • Confirmation bias leads to interpreting ambiguous answers as correct.

  • Model variability is high and a small sample doesn’t capture it.

Without structured measurement, the team develops improvements without knowing whether they improve anything. Worse, they may worsen without noticing until users complain.

The four key dimensions

RAG quality isn’t a single metric but a combination. Four dimensions capturing most problems:

Faithfulness: measures whether the response is supported by the retrieved context. A response can be plausible but contain information the context didn’t support: hallucinations.

Answer relevance: measures whether the answer effectively addresses the formulated question. A system can return correct but tangential answers to user intent.

Context precision: measures what fraction of retrieved context is actually relevant to the question. Retrieving a hundred documents of which only two are useful penalises the generator.

Context coverage: measures whether retrieved context contains all information needed to answer completely. Requires a reference answer to compare.

Frameworks like Ragas[1] automate calculation of these metrics using an LLM as judge, formalised in a 2023 paper[2]. TruLens[3] offers a similar view with variations. For starting teams, Ragas is a pragmatic entry point. Retrieval quality feeding these metrics depends directly on the strategy you chose: both your vector database choice and hybrid search versus pure vector have measurable impact on faithfulness and coverage.

Build a golden set

The most valuable asset in RAG evaluation isn’t the framework but the golden set: a carefully curated set of question-answer pairs representing real cases.

Starting point is real user logs if you have production. For new systems, interviews with target users and domain experts generate the most realistic questions. A typical golden set has between 100 and 500 questions, with deliberate coverage of:

  • Easy cases (direct questions with a clear answer).

  • Medium cases (require synthesis across several sources).

  • Hard cases (complex reasoning or scattered information).

  • Ambiguous cases (where the correct answer is debatable).

  • Out-of-scope cases (questions the system should not answer).

For each question, the golden set should include the correct answer (human-curated) and, optionally, the knowledge-base documents that should be retrieved. This last detail lets you measure retrieval recall independently of generation.

Golden set maintenance is a continuous activity. As the product evolves and new cases emerge, questions get added. The set is never complete, but it must stay representative.

LLM as judge: uses and limits

Many modern evaluations use an LLM as judge to score answers. It’s scalable and relatively cheap. But it has limitations:

  • The LLM judge tends to prefer longer answers, more formal, with more sophisticated structure.

  • Can be too lenient on certain subtle errors humans would catch.

  • If the same model is used for both generating and judging, bias is even higher.

Habitual mitigation involves:

  • Using a different model than the generator for judging.

  • Regularly validating with human sampling.

  • Using carefully designed evaluation prompts.

  • Using several LLM judges and averaging: consensus across three models reduces individual biases.

Never use only LLM as judge without periodic human validation. Monthly sampling of 50 human-reviewed answers, compared with automatic verdict, shows whether the judge is calibrated with reality.

Continuous evaluation in the pipeline

Once golden set is built and metrics defined, evaluation should integrate into development cycle. Every prompt change, every model change, every chunking change should pass through automated evaluation before merge.

The pattern that works in production includes thresholds on key metrics:

  • If faithfulness drops below 0.8, the build fails.

  • If answer relevance drops more than 5% against baseline, alert.

Thresholds are calibrated against historical data. Additionally, real traffic samples in production should pass through continuous evaluation, not just the golden set. This catches drift when users start phrasing questions outside the patterns originally anticipated.

Common mistakes

Patterns that repeat across projects:

  • Optimising metrics that don’t correlate with user satisfaction: a system can have high faithfulness but answers users perceive as useless. Validation with real users is irreplaceable.

  • Too-easy golden set: if the system scores 95% on all metrics but users complain, the set doesn’t capture real difficulty. Deliberately adding hard cases is healthy.

  • Not versioning the golden set with code: if the set changes without traceability, results can’t be compared across system versions. Git is enough.

Cost and time

Evaluating a set of 500 questions with a frontier LLM like GPT-4o costs between 10 and 50 euros per run, depending on answer length and the number of metrics. For frequent releases, adjust the set size or use cheaper models in daily CI, running the full set only on releases.

Evaluation time is real too. 500 questions with several metrics can take between 15 and 60 minutes. Parallelising is possible but not always trivial.

Complementary qualitative metrics

Quantitative metrics don’t capture everything. Important qualitative signals:

  • Explicit user feedback (thumbs up/down).

  • Time users spend with each answer.

  • Follow-up question rate.

  • Abandonment rate.

A system with excellent automatic metrics but a high user abandonment rate has a problem the metrics don’t capture. Combining both sources gives a complete picture.

Frequently asked questions

How many questions does a RAG golden set need?

Between 100 and 500 in a mature system, though a team just starting out can get useful signal from a minimum of 50 well-selected questions covering easy, medium, hard, ambiguous, and out-of-scope cases.

Is it safe to use an LLM as judge without human review?

Not exclusively. The LLM judge tends to prefer long, well-structured answers and can be too lenient with subtle errors; a monthly human sample (for example, 50 answers) is the validation that confirms whether the judge stays calibrated.

How much does it cost to evaluate a RAG system with a frontier LLM?

Evaluating a set of 500 questions with a model like GPT-4o costs between 10 and 50 euros per run, and the process can take between 15 and 60 minutes depending on the number of metrics.

Conclusion

Rigorously evaluating a RAG is the difference between a system that works well and one that seems to work well. Cost of implementing continuous evaluation is moderate; cost of not doing so is systems disappointing in production without the team knowing why. Tools like Ragas democratised infrastructure; the golden set and discipline of measuring consistently are now the hard part. For starting teams, a minimum set of 50 well-selected questions and the four basic metrics is already enormous leap over evaluating without structure.

Spanish version of this article.

Sources

  1. Ragas
  2. 2023 paper
  3. TruLens