promptfoo

Executes the same prompt across multiple LLM providers (OpenAI, Anthropic, Google, AWS Bedrock, Ollama, local models) in parallel, collecting structured outputs with metadata (latency, token counts, cost). Uses a provider registry pattern with pluggable provider implementations that normalize API differences into a unified interface, enabling side-by-side comparison of model behavior on identical inputs.

Defines test assertions (exact match, similarity, regex, LLM-based grading) that automatically evaluate whether model outputs meet criteria. Supports custom evaluator functions (JavaScript, Python, HTTP webhooks) that receive the prompt, output, and test case metadata, returning a pass/fail score and optional details. Assertions are composable and can be chained to create complex evaluation logic without writing test harnesses.

Stores evaluation results in local SQLite database or cloud storage (AWS S3, Google Cloud Storage, etc.), enabling historical tracking of prompt quality over time. Results include full metadata (prompt, model, variables, outputs, scores, latency, cost). Enables trend analysis (e.g., 'pass rate improved 5% over last month') and regression detection by comparing against previous baselines.

Provides native integration with AWS Bedrock (Claude, Llama, Mistral models), Google Vertex AI, Azure OpenAI, and other cloud providers. Handles authentication (IAM roles, API keys), model selection, and parameter mapping. Enables teams to test against cloud-hosted models without writing custom provider code. Supports streaming responses for real-time output evaluation.

Executes Python scripts (3.7+) and Node.js scripts (18+) as providers, passing prompt and variables as command-line arguments or stdin. Scripts can implement arbitrary logic (e.g., calling local models, preprocessing inputs, routing to multiple models). Output is captured from stdout and parsed as JSON or plain text. Enables teams to test custom inference logic without modifying promptfoo.

Provides native integration with Ollama (local LLM inference engine) and compatible local model servers (LLaMA.cpp, LocalAI). Connects to local HTTP endpoints, enabling teams to test open-source models (Llama, Mistral, etc.) without cloud API costs or latency. Supports model selection, parameter tuning, and streaming responses.

Provides CLI and web UI search/filtering capabilities to navigate large evaluation result sets. Supports filtering by test case name, provider, model, pass/fail status, and custom metadata. Search uses full-text indexing for fast queries. Enables teams to quickly find specific test cases or failure patterns without manually reviewing all results.

Generates adversarial test cases using attack strategies (jailbreaks, prompt injection, prompt leaking, toxicity, bias) to probe LLM vulnerabilities. Uses a plugin-based attack provider system where each strategy (e.g., 'crescendo jailbreak', 'SQL injection') generates variations of inputs designed to trigger unsafe behavior. Results are graded using guardrails (safety checks) to identify which attacks succeeded, producing a vulnerability report.

Defines test suites as YAML/JSON files with templated prompts, test cases, and variables. Supports variable substitution using {{variable}} syntax, allowing a single prompt template to be tested against multiple input combinations. Test cases can include expected outputs, assertions, and metadata. Configuration is declarative and version-controllable, enabling teams to track prompt changes over time.

Integrates evaluation results into CI/CD workflows via GitHub Actions, GitLab CI, or generic webhook triggers. Compares current evaluation results against baseline results to detect regressions (e.g., 'pass rate dropped from 95% to 90%'). Fails CI builds if regressions exceed configured thresholds, preventing degraded prompts from being merged. Results can be stored locally or uploaded to cloud storage for historical tracking.

Provides a local web interface (React-based frontend) for visualizing evaluation results, filtering by test case or provider, and comparing model outputs side-by-side. Results can be shared via shareable URLs (with optional cloud storage backend) or self-hosted. The UI supports real-time updates when new evaluation results are available, and includes search/filtering to navigate large result sets.

Supports custom providers via HTTP endpoints (POST requests with prompt/variables, returns output) and script execution (Python, Node.js, shell scripts). Allows teams to test against proprietary models, internal APIs, or custom inference servers without modifying promptfoo code. Scripts receive prompt and variables as arguments, execute locally, and return output to be graded.

Automatically tracks API costs per provider using model-specific pricing tables (OpenAI, Anthropic, Google, AWS, etc.), and measures latency for each API call. Aggregates costs and latency by provider, test case, and overall suite. Enables cost-benefit analysis (e.g., 'GPT-4 is 10x more expensive but only 5% more accurate'). Pricing tables are updated with each release to reflect current API costs.

Processes prompt templates with {{variable}} syntax, supporting variable substitution, array expansion (cartesian product of multiple variable values), and nested variable references. Allows a single prompt template to generate multiple test cases by expanding variables. Supports both simple string substitution and complex variable structures (objects, arrays).

Uses another LLM (OpenAI, Anthropic, Google, etc.) to grade model outputs against custom rubrics. Rubrics are defined as text descriptions of evaluation criteria (e.g., 'Is the response accurate? Is it helpful? Is it concise?'). The grading LLM receives the prompt, output, and rubric, and returns a score (0-1) and reasoning. Enables subjective quality evaluation without manual review.