Safety And Instruction Following Compliance Scoring

via “constraint compliance scoring and aggregation”

Google's benchmark for verifiable instruction following.

Unique: IFEval's scoring system supports multiple aggregation strategies and provides per-constraint breakdowns alongside aggregate scores, enabling both high-level performance comparison and diagnostic analysis of which constraint types cause failures.

vs others: Unlike single-metric evaluation approaches (e.g., accuracy), IFEval's multi-level scoring provides diagnostic granularity while still supporting simple aggregate comparisons, allowing researchers to understand both overall performance and specific failure modes.

via “safety and instruction-following compliance scoring”

Real-world user query benchmark judged by GPT-4.

Unique: Separates safety and instruction-following into independent scoring dimensions, revealing models that may be safe but non-compliant (or vice versa). Uses GPT-4 to evaluate nuanced safety concepts (appropriate refusal of harmful requests, absence of bias, ethical reasoning) rather than simple keyword filtering or rule-based detection.

vs others: More comprehensive than rule-based safety filters because it evaluates contextual safety and appropriate refusal; more practical than human safety review because it scales to 1,024 queries; more aligned with real-world safety concerns than synthetic adversarial benchmarks