Patronus AI

Evaluates LLM outputs for factual hallucinations using Patronus's proprietary Lynx 70B model, which performs semantic comparison between generated text and source documents to identify unsupported claims. The model operates via API calls priced at $10 per 1,000 evaluations for small evaluator instances, with results returned as structured scores and explanations. Integrates with the Patronus platform's experiment tracking system to log and compare hallucination rates across model versions.

Evaluates LLM outputs for harmful content including toxicity, offensive language, and brand safety violations using Patronus evaluator models. Scoring is delivered via API calls ($10-20 per 1,000 evaluations depending on evaluator size) with results integrated into the platform's experiment tracking and analytics dashboard. Supports comparison of toxicity rates across model versions and deployment environments.

Provides a REST API for programmatic evaluation of LLM outputs, with pricing based on evaluator size and evaluation type. Small evaluators cost $10 per 1,000 calls, large evaluators cost $20 per 1,000 calls, and evaluation explanations cost $10 per 1,000 calls. API calls are metered and billed monthly. The API integrates with the Patronus platform's experiment tracking and monitoring systems, enabling teams to build custom evaluation workflows.

Offers three subscription tiers (Individual free, Base $25/month, Enterprise custom) with different feature access and data retention policies. Free tier includes 2-week retention for Experiments, Logs, and Traces, plus unlimited Comparisons. Base tier adds analytics and reporting. Enterprise tier adds webhooks, on-prem/VPC deployment, custom data retention, and custom evaluation model fine-tuning. Feature access is enforced at the API and UI level.

Scans LLM outputs for personally identifiable information (PII) including names, email addresses, phone numbers, SSNs, and credit card numbers using pattern-matching and NLP-based detection. Results are returned via API with identified PII entities flagged and optionally redacted. Integrates with Patronus experiment tracking to monitor PII leakage rates across model versions and identify high-risk prompts or domains.

Generates adversarial prompts and test cases designed to expose weaknesses in LLM behavior, including jailbreak attempts, edge cases, and harmful instruction-following scenarios. The platform uses a combination of template-based prompt generation and learned adversarial patterns to create test suites that are executed against target models. Results are tracked in the Patronus Experiments system with detailed logs of which adversarial prompts succeeded in eliciting unsafe outputs.

Enables teams to define baseline evaluation metrics (hallucination rate, toxicity score, PII leakage, red-teaming results) and automatically compare new model versions or prompt changes against those baselines. The Patronus Comparisons feature provides side-by-side evaluation results with statistical significance testing and trend analysis. Results are persisted in the platform's experiment tracking system with unlimited retention on paid tiers.

Monitors LLM outputs in production environments in real-time, tracking hallucination rates, toxicity scores, PII leakage, and other safety metrics across time. The Patronus Logs feature captures evaluation results for all production queries, while the Patronus Traces feature provides detailed execution traces. Analytics dashboards aggregate metrics by time period, user segment, or prompt category, enabling teams to detect safety regressions or anomalies in production behavior.

Stores and organizes evaluation runs as 'experiments' with metadata including model version, prompt, dataset, and evaluation results. Each experiment is timestamped and can be compared against other experiments. The platform provides a searchable experiment history with 2-week retention on free tier and unlimited retention on paid tiers. Experiments can be tagged, annotated, and organized into projects for team collaboration.

Provides pre-built simulation environments (research science, software development, customer service, product applications, finance) where agents can be trained and evaluated. Each environment includes domain-specific datasets, reward functions, and evaluation metrics. The platform hosts 1M+ world data artifacts contributed by 5,000+ expert contributors, enabling realistic agent training scenarios. Agents interact with simulated environments to develop behaviors, with performance tracked against domain-specific benchmarks.

Provides curated benchmark datasets for evaluating agent and model performance in specific domains. Named benchmarks include FinanceBench (10,000 Q&A pairs for financial domain), BLUR (573 Q&A pairs for unknown domain), and others. Benchmarks are designed to test specific capabilities (e.g., financial reasoning, domain knowledge) and are used to evaluate both agents in simulation and LLM outputs. Benchmark results are comparable across models and versions.

Hosts and distributes proprietary research models including Lynx (70B hallucination detection model), GLIDER (evaluation model), and others. Models are made available via API for evaluation tasks, with pricing based on model size and evaluation complexity. Models are documented in published research papers and can be cited in academic work. The platform provides version tracking and ensures reproducibility of published results.