Experiment Logging And Result Persistence With Structured Output

via “test result persistence and historical comparison”

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

Unique: Uses config hash-based matching to automatically correlate results across runs, enabling trend analysis without manual baseline management. Stores full result details (responses, assertion outcomes) enabling post-hoc analysis and debugging of historical test runs.

vs others: More convenient than manual result tracking because historical data is automatically persisted, and more actionable than single-run results because trend analysis reveals whether changes improved or degraded quality.