promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simpl
What is promptfoo?
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Overview of promptfoo
What it is
Promptfoo is a CLI and library for evaluating and red-teaming LLM-based applications. It is available as an open source project under the MIT license. The tool supports automated evaluations, vulnerability scanning, and side-by-side model comparisons. It is used by OpenAI and Anthropic. Promptfoo is now part of OpenAI but remains open source and MIT licensed.
Key features
- Test prompts and models with automated evaluations
- Secure LLM applications with red teaming and vulnerability scanning
- Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
- Automate checks in CI/CD
- Review pull requests for LLM-related security and compliance issues with code scanning
- Share results with your team
Use cases
Promptfoo is designed for developers building LLM-based apps who need to move from trial-and-error to data-driven decisions. Use cases include evaluating prompt quality, red teaming for security vulnerabilities, comparing model performance across providers, integrating automated checks into CI/CD pipelines, and reviewing pull requests for security and compliance issues.
Getting started
shnpm install -g promptfoopromptfoo init --example getting-startedAlso available via brew install promptfoo and pip install promptfoo. Or use npx promptfoo@latest to run any command without installing. Most LLM providers require an API key set as an environment variable (e.g., export OPENAI_API_KEY=sk-abc123). Run promptfoo eval and promptfoo view to run an eval and view results.
License and pricing
Promptfoo is open source under the MIT license. Pricing type is open source.
Based on the project's public README and repository metadata.