promptfoo is an open-source tool to test, evaluate and attack (red team) prompts, agents and RAG systems from declarative configuration files. Instead of eyeballing whether a prompt change improves the answers, you define test cases in YAML, run an evaluation and get a matrix that compares models and versions with automatic scores. In this guide you will see what promptfoo is, how test cases are declared, how to compare models, what its red teaming mode does and how to wire it into your CI. The same explanation is available in Spanish.

Key takeaways

  • promptfoo is an open-source command-line tool (MIT licence) written for Node.js; it has over 22,000 stars on GitHub and is on version 0.121.19, released on 14 July 2026.
  • In March 2026 OpenAI acquired it: the announcement is dated 9 March 2026 and the project stays open source under the same licence, so you can use it for free and run it on your own machine.
  • You work with declarative YAML configs: you define prompts, providers (over 60 providers: OpenAI, Anthropic, Gemini, DeepSeek and local models with Ollama) and tests with assertions that score each answer.
  • Its red teaming mode generates adversarial prompts for over 50 vulnerability types (prompt injection, jailbreaks, data leakage, excessive agency) with presets for OWASP LLM Top 10, NIST AI RMF and MITRE ATLAS.
  • It plugs into CI: promptfoo eval returns a non-zero exit code when assertions fail, turning your prompt tests into a quality gate like any unit test.

What is promptfoo?

promptfoo is a developer tool to test and evaluate applications built on language models: standalone prompts, RAG chains and full AI agents. The core idea is to treat prompts the way you treat code: with repeatable tests instead of subjective impressions. The official documentation sums it up as "test and evaluate your prompts, models and RAG pipelines".

The project started in 2024, built by Ian Webster and Michael D’Angelo, and its adoption has been notable: more than 350,000 developers use it, around 130,000 active each month, and over a quarter of Fortune 500 companies. Its own repository boasts of being "used by OpenAI and Anthropic". In March 2026, OpenAI announced its acquisition; in OpenAI’s own words, "promptfoo will remain open source under the current license". It still ships under the MIT licence and is on version 0.121.19.

You install it with Node.js in a single command, and it needs no account or server: everything runs locally against the model APIs you already use.

# Global install
npm install -g promptfoo

# Or without installing anything, bootstrapping an example
npx promptfoo@latest init --example getting-started

How are test cases declared?

Everything revolves around a promptfooconfig.yaml file with three blocks. In prompts you write the templates, with variables in double braces. In providers you list the models you want to test. And in tests you define each case: the variable values and the assertions that decide whether the answer passes or fails.

prompts:
  - "Translate to {{language}}: {{text}}"

providers:
  - openai:gpt-4o
  - anthropic:messages:claude-sonnet-4

tests:
  - vars:
      language: French
      text: Hello world
    assert:
      - type: icontains
        value: "Bonjour"
      - type: llm-rubric
        value: The translation is natural and correct

Assertions are the key. Some are deterministic (contains, icontains, equals, is-json, cost, latency) and some are model-graded, like llm-rubric, which judges an answer against a natural-language criterion. You can also write custom assertions in javascript or python. For an agent that calls tools, this lets you check not just the final text but that the right tool was invoked with the right arguments.

How do you compare models and prompts?

Here is the tool’s daily value. When you run the evaluation, promptfoo tests every combination of prompt, provider and case, and lays the results out in a matrix: rows for the test cases, columns for each model or prompt variant, and in each cell the answer with its pass or fail verdict.

# Run the evaluation defined in promptfooconfig.yaml
npx promptfoo eval

# Open the results table in the browser
npx promptfoo view

That side-by-side view answers the question everyone asks: is it worth paying for the big model, or does the small one give equally good answers at a lower cost? Since each cell also shows cost and latency, the decision stops being a hunch and starts leaning on numbers. It is the same job an evaluation tool like DeepEval does, but promptfoo aims it at fast comparison between models and prompts from a single file.

What is red teaming and vulnerability scanning?

Beyond measuring quality, promptfoo attacks your system to find security flaws before a malicious user does. Its red teaming mode automatically generates adversarial prompts covering over 50 vulnerability types: prompt injection, jailbreaks, personal-data leakage, unauthorised function access (BOLA and BFLA) and excessive agent agency, among others. It ships presets aligned with standards like the OWASP LLM Top 10, the NIST AI RMF framework and MITRE ATLAS, and even a multi-turn strategy called Hydra that adapts the attack based on the system’s responses.

# Set up the scan (generates a red team config)
promptfoo redteam init --no-gui

# Launch the attack, then open the report
npx promptfoo redteam run
npx promptfoo redteam report

Having this capability live in the same tool as the quality tests makes sense: an agent with tool access is an attack surface and deserves to be treated as one. It is no coincidence that OpenAI acquired promptfoo precisely to strengthen the security testing of its agents.

How do you integrate it into CI?

Because promptfoo is a terminal command that returns an exit code, it fits naturally into any continuous-integration pipeline. If an assertion fails, promptfoo eval exits with a non-zero code and the pipeline run stops, just like a broken unit test. That keeps a prompt change that worsens the answers from reaching production.

The project publishes an official GitHub Action that runs the evaluation on every pull request and comments the results straight into the thread. You can also invoke promptfoo eval from GitLab CI, Jenkins or any other system. Combined with an observability tool like Langfuse, you get a full loop: tests before deploying and traces afterwards.

Frequently asked questions

How does promptfoo differ from DeepEval?

Both evaluate LLM applications, but with different approaches. promptfoo is a command-line tool centred on declarative YAML configs, comparing models in a matrix and security red teaming; it shines for iterating fast on prompts. DeepEval is a Python library in the style of pytest, meant for writing evaluation metrics as tests inside your code. If you live in the terminal and want to compare prompts and models quickly, promptfoo fits better; if you want your evaluations as Python code next to your tests, DeepEval is more natural.

Do I need an account or need to pay to use promptfoo?

No. promptfoo is open source under the MIT licence and runs locally on your machine; you need no account or company server. You only pay for the API calls to the models you test, and if you use an open model with Ollama, not even that. OpenAI’s March 2026 acquisition did not change the licence: it stays free.

Can promptfoo test agents and not just standalone prompts?

Yes. You can point promptfoo at a custom provider that wraps your full agent, with its tools and memory, and evaluate the final output. Assertions let you check both the answer text and the intermediate behaviour, for example whether the right tool was called. For an agent like the ones you build with the Anthropic SDK, this turns each task into a reproducible test case.

Conclusion

promptfoo brings the discipline of automated testing to prompts and AI agents: declarative cases in YAML, model comparison in a table, assertions that score each answer and a red teaming mode that hunts for vulnerabilities before anyone else does. That it is open source under the MIT licence, that it runs locally and that OpenAI acquired it in 2026 give a sense of its solidity. The next step is to install it with npm install -g promptfoo, write your first promptfooconfig.yaml with two models and a handful of cases, and let promptfoo eval decide for you which one answers best.

Sources: [1] Official promptfoo documentation[1], [2] promptfoo on GitHub[2], [3] OpenAI announces the acquisition of promptfoo[3], [4] CNBC coverage of the acquisition[4].

Visitas: 0

Sources

  1. Official promptfoo documentation
  2. promptfoo on GitHub
  3. OpenAI announces the acquisition of promptfoo
  4. CNBC coverage of the acquisition

Route: AI Agents in Production: Evaluation and Reliability