DeepEval: evaluating AI agents

DeepEval is the most widely used open-source framework for evaluating and testing AI systems, and its great insight is that you drive it just like Pytest: instead of eyeballing whether an answer "sounds right", you write test cases, attach metrics to them and let the framework score each output from 0 to 1. If you have already decided what to measure in an agent and want a concrete tool to measure it with, this is the go-to library. In this guide you will see what DeepEval is, its main metrics, how to evaluate an agent and its tools, how to wire it into your CI with pytest and how it differs from promptfoo. The same explanation is available in Spanish.

Key takeaways

• DeepEval, from the company Confident AI, is an open-source framework for evaluating LLMs and agents; it has over 16,900 stars on GitHub and is used to run more than 20 million evaluations a day.
• It is on version 4.1.1, released on 16 July 2026, requires Python 3.9 or higher and installs with a single pip install -U deepeval.
• It offers more than 50 research-backed metrics: G-Eval, faithfulness, answer relevancy, tool correctness and task completion, among others.
• The metrics run on your own machine and every score ranges from 0 to 1, with a default threshold of 0.5: above it passes, below it fails.
• You write it like a pytest test (deepeval test run), so it slots naturally into continuous integration to catch regressions on every change.

What is DeepEval?

DeepEval is a Python library built by Confident AI that treats every model or agent response as something you can assert, score and gate in your CI. The official documentation puts it plainly: "it is similar to Pytest but specialized for testing language-model outputs". That comparison is no accident: if you have ever written a unit test, DeepEval will feel familiar from the first minute.

It is worth placing it. Deciding what makes an agent reliable, by what criteria to judge it and how to assemble the test suite is a conceptual problem we cover separately. DeepEval is the practical answer to which tool do I measure it with: the concrete library, installable with pip, that turns those criteria into runnable metrics. Its basic unit is the LLMTestCase, an object holding the input, the system’s actual output and, when available, the expected output and the retrieved context.

pip install -U deepeval

What metrics does it offer: relevancy, faithfulness and G-Eval?

Here is the heart of DeepEval. A metric receives a test case and returns a score between 0 and 1; if it meets or beats the threshold (0.5 by default), the test passes. The library groups more than fifty metrics into families by what you are evaluating:

• RAG metrics: answer relevancy, faithfulness (that the output does not contradict the retrieved context), and contextual precision, recall and relevancy.
• Agent metrics: tool correctness (did it call the right tools?), task completion, argument correctness and step efficiency.
• Conversational metrics: knowledge retention, role adherence and conversation completeness in multi-turn dialogues.
• Safety metrics: bias, toxicity, PII leakage and misuse.

The star is G-Eval: an LLM-as-a-judge metric where you define the criterion in natural language ("check that the answer is correct and omits no data from the reference text"), and DeepEval generates the evaluation steps on its own and asks a judge model to score the output. For criteria that demand a deterministic decision there is DAG, a decision-tree-based judge.

from deepeval import evaluate
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

correctness = GEval(
    name="Correctness",
    criteria="Check whether the actual output answers the input without making up data.",
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    threshold=0.5,
)

case = LLMTestCase(
    input="What is the capital of France?",
    actual_output="The capital of France is Paris.",
)

evaluate(test_cases=[case], metrics=[correctness])

One important detail: the metrics run on your own machine. You choose the judge model (OpenAI, Gemini, Azure or one you run locally with Ollama), so none of your responses need ever leave your infrastructure.

How do you evaluate an agent and its tools?

An agent does not just produce text: it reasons, decides and calls tools. DeepEval ships metrics built for that. Tool Correctness compares the tools the agent actually used against the ones it should have used, and Task Completion judges whether the user’s goal was met by the end of the run. Because this kind of metric does not always need a predefined ideal answer (they are referenceless), it works even for evaluating in production, where you rarely have the correct answer to hand.

from deepeval.metrics import ToolCorrectnessMetric
from deepeval.test_case import LLMTestCase, ToolCall

case = LLMTestCase(
    input="What's the weather in Madrid?",
    actual_output="Right now it's 28 degrees and clear.",
    tools_called=[ToolCall(name="get_weather")],
    expected_tools=[ToolCall(name="get_weather")],
)

metric = ToolCorrectnessMetric()
metric.measure(case)
print(metric.score)   # 1.0 if it called the expected tool

These agent metrics pair well with the reflection pattern: you can measure whether the self-critique phase truly improves the completion score or merely adds token cost with no real gain.

How does it integrate into CI with pytest?

That is DeepEval’s selling point: an evaluation is not a notebook you run by hand once, but a test that runs on every commit. You write the test as if it were pytest, use assert_test to require the metrics to pass and launch it with DeepEval’s own runner.

import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_relevancy():
    case = LLMTestCase(
        input="Summarise the article in one sentence.",
        actual_output="The article explains how to evaluate agents with DeepEval.",
    )
    assert_test(case, [AnswerRelevancyMetric(threshold=0.7)])

deepeval test run test_agent.py

If any metric falls below its threshold, the command fails with a non-zero exit code, exactly as pytest would; your GitHub Actions or GitLab CI pipeline marks the build red and stops a regression before it reaches production. This evaluation-as-code approach it shares with optimisation tools like DSPy, which uses similar metrics to compile prompts rather than just watch them.

DeepEval versus promptfoo

DeepEval and promptfoo solve the same problem, measuring the quality of an AI system, from opposite philosophies, and choosing well depends on your team.

In practice: if your agent already lives in Python and you want rich metrics and tests that fail in CI, DeepEval fits like a glove. If you would rather describe dozens of prompt-and-model combinations in a YAML and compare them without writing code, promptfoo is nimbler. They are not mutually exclusive: some teams use promptfoo to explore and DeepEval to lock down in CI what already works.

Frequently asked questions

Does DeepEval need to send my data to a cloud?

No. The metrics run on your own machine and you choose the judge model, which can be a local one served by Ollama. Confident AI also offers a cloud platform (dashboards, traces and datasets) built on DeepEval, but it is optional: the library works fully without it.

Which model acts as the judge in G-Eval?

The one you configure. By default DeepEval uses an OpenAI model, but it supports Azure, Gemini, local models via Ollama and custom judges. Choosing a capable judge matters: G-Eval inherits both the reliability and the biases of the model that scores, so a strong model is advisable for subjective metrics.

Is it for evaluating in production and not just in tests?

Yes. Many metrics are referenceless, meaning they do not require a predefined ideal answer, so they can score real user responses online. Even so, the most common use is in CI: deterministic tests with fixed cases that stop a regression before you deploy.

Conclusion

DeepEval turns agent evaluation into something as routine as writing a unit test: you define cases, pick from more than fifty research-backed metrics, get scores from 0 to 1 and gate any regression in your CI, all without your data leaving your machine. With over 16,900 stars and 20 million evaluations a day, it is the de facto standard for testing AI systems in Python. The next step is to install it with pip install -U deepeval, write your first G-Eval metric and compare it with promptfoo to decide which fits your workflow best.

Sources: [1] Official DeepEval documentation^[1], [2] DeepEval on GitHub (Confident AI)^[2], [3] deepeval package on PyPI^[3], [4] Confident AI, the company behind DeepEval^[4].

Sources

1. Official DeepEval documentation
2. DeepEval on GitHub (Confident AI)
3. deepeval package on PyPI
4. Confident AI, the company behind DeepEval

Route: AI Agents in Production: Evaluation and Reliability