MATH Benchmark

Loads and preprocesses 12,500 curated competition mathematics problems from AMC 10/12, AIME, and Math Olympiads using the MATHDataset class in MATH.py. The loader supports multiple tokenization strategies and can selectively include or exclude solution steps during preprocessing, enabling researchers to evaluate models on problem-solving without solution hints. Problems are stratified across 7 mathematical subjects (Prealgebra, Algebra, Number Theory, Counting/Probability, Geometry, Intermediate Algebra, Precalculus) with structured JSON metadata including problem statements, solutions, and difficulty levels.

Implements the is_equiv() function in math_equivalence.py that determines semantic equivalence between two mathematical expressions regardless of syntactic representation. The system applies a multi-stage normalization pipeline that handles LaTeX formatting, fraction representations, algebraic simplification, and numerical precision issues before performing string-based comparison. This enables accurate answer verification without requiring exact string matching, accommodating equivalent forms like '1/2', '0.5', and '\frac{1}{2}'.

Extracts and preserves solution steps from MATH problems, enabling evaluation of intermediate reasoning and chain-of-thought capabilities. The system can optionally include or exclude solution steps during dataset loading, supporting different evaluation methodologies: evaluating final answers only (without hints) or evaluating intermediate reasoning steps. This enables researchers to assess whether models can generate correct reasoning chains or merely guess final answers.

Provides evaluation infrastructure in eval_math_gpt.py that runs local language models (GPT-style architectures) on MATH dataset problems with configurable inference parameters including beam search width, sampling temperature, and top-k/top-p filtering. The run_eval() function orchestrates the evaluation pipeline: loads problems from MATHDataset, generates model responses with specified decoding strategy, extracts final answers from model outputs, and compares against ground truth using mathematical equivalence checking. Supports both greedy decoding and stochastic sampling for exploring model behavior under different inference regimes.

Provides evaluation infrastructure in evaluate_gpt3.py that interfaces with OpenAI's GPT-3 API for remote model evaluation on MATH problems. The system handles API authentication, batches problem submissions to the GPT-3 API, parses structured responses, and aggregates accuracy metrics. This enables evaluation of closed-source models without local compute resources, though with latency and cost considerations inherent to API-based inference.

Aggregates evaluation results across the 12,500 problems and computes accuracy metrics stratified by mathematical subject (Prealgebra, Algebra, Number Theory, Counting/Probability, Geometry, Intermediate Algebra, Precalculus). The reporting system generates per-subject accuracy percentages, overall accuracy, and optional per-difficulty breakdowns. This enables fine-grained analysis of model strengths and weaknesses across mathematical domains, revealing whether models struggle with specific subject areas.

Extracts and indexes structured metadata from MATH dataset JSON files including problem statement, solution steps, final answer, difficulty level, and mathematical subject. The indexing system enables efficient retrieval of problems by subject, difficulty, or other attributes, and provides structured access to problem components (problem text vs solution vs answer) for different evaluation workflows. Metadata is preserved throughout the evaluation pipeline to enable stratified analysis and filtering.

Extracts final numerical or symbolic answers from model-generated text using heuristic pattern matching (e.g., regex patterns for 'Answer: X', 'Final Answer:', or boxed notation). The extraction system handles common answer formats including integers, fractions, decimals, and algebraic expressions. This enables automatic answer verification without requiring models to output structured JSON or follow strict formatting conventions, accommodating natural language model outputs.

Curates and distributes the MATH dataset of 12,500 problems sourced from official mathematical competitions (AMC 10, AMC 12, AIME, and Math Olympiads). Problems are manually collected, verified for correctness, and formatted into standardized JSON structure with problem statement, solution, and metadata. The dataset is hosted on Berkeley servers and distributed via GitHub repository, enabling researchers to access high-quality competition mathematics problems for benchmarking.

Annotates each MATH problem with a difficulty level (easy, medium, hard) based on competition source and problem characteristics. The stratification system enables evaluation of model performance across difficulty tiers, revealing whether models struggle more with harder problems or show consistent performance. Difficulty annotations are preserved in problem metadata and used for stratified accuracy reporting.

Provides access to the AMPS (Algebraic Mathematical Problem Solving) pretraining dataset, a large-scale collection of mathematical problems designed for pretraining language models on mathematical reasoning. The integration enables researchers to use AMPS for model pretraining and then evaluate the pretrained models on MATH benchmark, creating a complete pipeline from pretraining to evaluation. AMPS dataset is hosted on Google servers and can be downloaded separately from MATH.