Response Filtering And Answer Extraction With Regex And Parsing

EleutherAI's evaluation framework — 200+ benchmarks, powers Open LLM Leaderboard.

Unique: Provides a pluggable filter system where each task can define custom extraction logic via regex, JSON parsing, or Python functions. Filters are applied in sequence with fallback strategies, allowing graceful degradation if primary extraction fails. The system logs extraction failures for debugging and supports multiple valid answer formats.

vs others: Supports multiple extraction strategies with fallbacks, whereas alternatives typically use single-strategy extraction; integrates extraction into the evaluation pipeline rather than requiring post-processing

via “answer extraction from model outputs with heuristic parsing”

12.5K competition math problems — AMC/AIME/Olympiad level, 7 subjects, standard math benchmark.

Unique: Uses lightweight regex-based heuristics rather than requiring models to output structured JSON, enabling evaluation of base language models without answer format fine-tuning. This pragmatic approach trades robustness for flexibility, accommodating diverse model output styles.

vs others: More flexible than requiring structured output because it works with any model without fine-tuning, but less reliable than models trained to output answers in standardized formats (e.g., JSON with 'answer' field).