Retrieval Evaluation Frameworks: Ragas and Similar

Building a RAG system is relatively easy: embeddings + vector DB + LLM. Measuring whether it works well is the real challenge. There are three questions to answer: whether answers are faithful to the retrieved context, whether the context is relevant, and whether the answer addresses the question.

Without metrics, evaluation is intuition. Ragas^[1] and similar frameworks turn those questions into numbers comparable across versions.

Key takeaways

• Ragas defines four core metrics: faithfulness, answer_relevancy, context_precision, and context_recall.

• Faithfulness detects hallucinations: what fraction of answer claims are supported by the retrieved context.

• Integrating evaluation in CI from day one detects regressions before they reach production.

• Evaluating with GPT-4 has significant cost: subset mode and cheaper evaluators reduce spending.

• Metrics are a proxy: periodic human review remains the ground truth.

The four metrics that matter

Faithfulness

Is the answer backed by the retrieved context? It is calculated as the fraction of claims in the answer that can be derived from the context. Low faithfulness signals hallucination: the model is inventing statements not present in the retrieved documents.

Answer Relevancy

Does the answer address the original question? It is evaluated by generating hypothetical questions from the answer and comparing them with the original question. A high score means the answer targets the right intent.

Context Precision

Of the retrieved context, what fraction is relevant to the question? It penalises noisy retrieval. Useful for tuning chunk size and the top-k parameter.

Context Recall

Does the retrieved context contain all the information needed to answer correctly? It requires a ground-truth answer. It detects when retrieval leaves important information out.

Basic Ragas usage

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

data = {
    "question": ["What is RAG?", "What are its advantages?"],
    "answer": ["RAG combines retrieval with generation.", "Less hallucination..."],
    "contexts": [
        ["RAG (Retrieval Augmented Generation) combines..."],
        ["The advantages of RAG include..."]
    ],
    "ground_truth": ["RAG is a technique...", "Main advantages..."]
}

dataset = Dataset.from_dict(data)
result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)
print(result)

Ragas uses OpenAI by default, and you can point it at any LLM. For retrieval itself, see nomic-embed-text for open embeddings and pgvector for the vector index.

CI integration

The goal is catching regressions whenever you change prompts, models, chunk size, or retrieval strategy:

# .github/workflows/rag-eval.yml
- name: Evaluate RAG
  run: |
    python evaluate.py --threshold-faithfulness 0.8

# evaluate.py
results = evaluate(dataset, metrics=[faithfulness])
if results["faithfulness"] < 0.8:
    sys.exit(1)  # Fails CI, blocks merge

Thresholds are set from the historical baseline. A change that drops faithfulness by 10% is a red flag. Integrating evaluation in CI from day one is an investment that pays dividends: it catches regressions before production and gives the system credibility with stakeholders.

Ragas alternatives

Each framework has its own specific focus:

TruLens^[2]: similar metrics plus a triad (context_relevance, groundedness, answer_relevance). It ships a built-in web dashboard and has strong LangChain integration. A good option if you already use LangChain as your orchestrator; see LangGraph for agent workflows.

DeepEval^[3]: a pytest-like framework with custom metrics you add as plain functions. Ready-to-use CI/CD integration. The closest to the software-testing workflows teams already know.

Giskard^[4]: RAG evaluation plus security and bias detection. Commercial tier with a free tier. Useful when safety and fairness are explicit requirements.

Arize Phoenix^[5]: observability for LLM applications, including evaluation. Open source with a SaaS option.

LLM as judge

These frameworks use an LLM (GPT-4 by default) to calculate the metrics. That has implications:

• Advantage: it scales, and it can evaluate multiple facets at once.

• Disadvantage: the judge has its own biases, the cost is real, and results are not deterministic.

Emerging improvements:

• Small fine-tuned LLMs as evaluators: cheaper and more reproducible.

• Multi-LLM consensus: three models vote and the majority wins.

• Periodic human review: to validate that the automatic metrics are still useful.

Cost management

Evaluating 500 questions with GPT-4 across four metrics implies roughly 6,000 LLM calls (about $60-300 per full evaluation run). Strategies to keep it under control:

• Subset mode in CI: 50 questions per PR, full evaluation only on releases.

• Cheaper evaluator for the bulk of the work (GPT-4o mini), GPT-4 for the critical cases.

• Cache results: if the dataset and the model do not change, reuse the previous evaluation.

The evaluation dataset

The most valuable asset is not the tools but the evaluation dataset:

• Extract real questions from production user logs.

• Annotate ground truth with domain experts.

• 50-500 examples is reasonable to start.

• Include edge cases: ambiguous questions, out-of-scope, multi-step.

Tools are cheap. A representative, curated, maintained dataset is the difference between metrics that detect real problems and metrics that give false confidence.

Conclusion

Evaluating RAG rigorously is the difference between a system that "works in the demo" and one that "works in production". Ragas offers standard metrics with accessible implementation. TruLens, DeepEval and Giskard are valid alternatives with different focuses.

Building your own representative dataset is the most valuable asset. Integrating evaluation in CI from day one is the investment with the best return in any serious RAG project.

This article is also available in Spanish: Frameworks de evaluación para retrieval: Ragas y similares.

Frequently asked questions

How much does evaluating a RAG system with Ragas cost, and how do I reduce it?

Evaluating 500 questions with GPT-4 across the four metrics implies roughly 6,000 LLM calls, about $60-300 per full run. To keep it under control: subset mode in CI with 50 questions per PR and a full evaluation only on releases. Then a cheaper evaluator such as GPT-4o mini for the bulk of the work while reserving GPT-4 for critical cases, and caching results when the dataset and model do not change.

How do I detect that a prompt or chunk-size change has degraded my RAG?

By integrating evaluation in CI: a script that runs evaluate on the dataset and calls sys.exit(1) if faithfulness drops below a threshold, for example 0.8, blocks the merge. Thresholds are set from the historical baseline, and a 10% drop in faithfulness is a red flag. Doing this from day one catches regressions before production.

Do I need ground-truth answers to use the Ragas metrics?

Only for context_recall, which checks whether the retrieved context contains all the information needed and therefore requires a reference answer. Faithfulness measures the fraction of answer claims supported by the context, answer_relevancy generates hypothetical questions from the answer and compares them with the original. In turn, context_precision measures what fraction of the retrieved context is relevant. A dataset of 50-500 examples annotated by domain experts is reasonable to start.

Sources

1. Ragas
2. TruLens
3. DeepEval
4. Giskard
5. Arize Phoenix