Updated: 2026-07-07

Building a RAG system is relatively easy: embeddings + vector DB + LLM. Measuring whether it works well is the real challenge. Are answers faithful to the retrieved context? Is the context relevant? Does the answer address the question? Without metrics, evaluation is intuition. Ragas[1] and similar frameworks turn those questions into numbers comparable across versions.

Key takeaways

  • Ragas defines four core metrics: faithfulness, answer_relevancy, context_precision, and context_recall.

  • Faithfulness detects hallucinations: what fraction of answer claims are supported by the retrieved context.

  • Integrating evaluation in CI from day one detects regressions before they reach production.

  • Evaluating with GPT-4 has significant cost: subset mode and cheaper evaluators reduce spending.

  • Metrics are a proxy: periodic human review remains the ground truth.

The four metrics that matter

Faithfulness

Is the answer backed by the retrieved context? It is calculated as the fraction of claims in the answer that can be derived from the context. Low faithfulness signals hallucination: the model is inventing statements not present in the retrieved documents.

Answer Relevancy

Does the answer address the original question? It is evaluated by generating hypothetical questions from the answer and comparing them with the original question. A high score means the answer targets the right intent.

Context Precision

Of the retrieved context, what fraction is relevant to the question? It penalises noisy retrieval. Useful for tuning chunk size and the top-k parameter.

Context Recall

Does the retrieved context contain all the information needed to answer correctly? It requires a ground-truth answer. It detects when retrieval leaves important information out.

Basic Ragas usage

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

data = {
    "question": ["What is RAG?", "What are its advantages?"],
    "answer": ["RAG combines retrieval with generation.", "Less hallucination..."],
    "contexts": [
        ["RAG (Retrieval Augmented Generation) combines..."],
        ["The advantages of RAG include..."]
    ],
    "ground_truth": ["RAG is a technique...", "Main advantages..."]
}

dataset = Dataset.from_dict(data)
result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)
print(result)

Ragas uses OpenAI by default but is configurable with any LLM. For retrieval itself, see nomic-embed-text for open embeddings and pgvector for the vector index.

CI integration

The goal is catching regressions whenever you change prompts, models, chunk size, or retrieval strategy:

# .github/workflows/rag-eval.yml
- name: Evaluate RAG
  run: |
    python evaluate.py --threshold-faithfulness 0.8
# evaluate.py
results = evaluate(dataset, metrics=[faithfulness])
if results["faithfulness"] < 0.8:
    sys.exit(1)  # Fails CI, blocks merge

Thresholds are set from the historical baseline. A change that drops faithfulness by 10% is a red flag. Integrating evaluation in CI from day one is an investment that pays dividends: it catches regressions before production and gives the system credibility with stakeholders.

Ragas alternatives

Each framework has its own specific focus:

TruLens[2]: similar metrics plus a triad (context_relevance, groundedness, answer_relevance). It ships a built-in web dashboard and has strong LangChain integration. A good option if you already use LangChain as your orchestrator; see LangGraph for agent workflows.

DeepEval[3]: a pytest-like framework with custom metrics that are easy to add. Ready-to-use CI/CD integration. The closest to the software-testing workflows teams already know.

Giskard[4]: RAG evaluation plus security and bias detection. Commercial tier with a free tier. Useful when safety and fairness are explicit requirements.

Arize Phoenix[5]: observability for LLM applications, including evaluation. Open source with a SaaS option.

LLM as judge

Most frameworks use an LLM (typically GPT-4) to calculate the metrics. That has implications:

  • Advantage: it scales, and it can evaluate multiple facets at once.

  • Disadvantage: the judge has its own biases, the cost is real, and results are not deterministic.

Emerging improvements:

  • Small fine-tuned LLMs as evaluators: cheaper and more reproducible.

  • Multi-LLM consensus: three models vote and the majority wins.

  • Periodic human review: to validate that the automatic metrics are still useful.

Cost management

Evaluating 500 questions with GPT-4 across four metrics implies roughly 6,000 LLM calls (about $60-300 per full evaluation run). Strategies to keep it under control:

  • Subset mode in CI: 50 questions per PR, full evaluation only on releases.

  • Cheaper evaluator for the bulk of the work (GPT-4o mini), GPT-4 for the critical cases.

  • Cache results: if the dataset and the model do not change, reuse the previous evaluation.

The evaluation dataset

The most valuable asset is not the tools but the evaluation dataset:

  • Extract real questions from production user logs.

  • Annotate ground truth with domain experts.

  • 50-500 examples is reasonable to start.

  • Include edge cases: ambiguous questions, out-of-scope, multi-step.

Tools are cheap. A representative, curated, maintained dataset is the difference between metrics that detect real problems and metrics that give false confidence.

Conclusion

Evaluating RAG rigorously is the difference between a system that "works in the demo" and one that "works in production". Ragas offers standard metrics with accessible implementation. TruLens, DeepEval and Giskard are valid alternatives with different focuses. Building your own representative dataset is the most valuable asset. Integrating evaluation in CI from day one is the investment with the best return in any serious RAG project.

This article is also available in Spanish: Frameworks de evaluación para retrieval: Ragas y similares.

Sources

  1. Ragas
  2. TruLens
  3. DeepEval
  4. Giskard
  5. Arize Phoenix