ZeroEval

Evaluates LLM performance on mathematical reasoning tasks without few-shot examples by implementing standardized problem sets with automated answer extraction and numerical correctness verification. Uses pattern-based answer parsing to handle diverse output formats (natural language, LaTeX, symbolic notation) and compares against ground-truth solutions with tolerance thresholds for floating-point accuracy.

Assesses LLM performance on formal logical reasoning tasks using standardized problem sets that require multi-step deduction without examples. Implements structured evaluation of premise-conclusion relationships with support for propositional logic, first-order logic, and natural language reasoning puzzles, using symbolic verification or semantic similarity matching to validate logical correctness.

Evaluates LLM code generation capability on programming tasks without few-shot examples using standardized problem sets with automated code execution and correctness verification. Implements test case execution against generated code with support for multiple programming languages, timeout handling, and detailed error reporting to distinguish between syntax errors, runtime failures, and logic errors.

Provides standardized dataset loading and management infrastructure for mathematical, logical, and code generation tasks with consistent problem formatting, answer key handling, and metadata tracking. Implements dataset versioning, problem filtering by difficulty/category, and batch processing support to enable reproducible evaluation across different problem domains with a single interface.

Orchestrates evaluation of multiple LLMs against benchmark datasets with support for different inference APIs (OpenAI, Anthropic, local models) and configurable inference parameters. Implements batch processing, result aggregation, and comparative analysis across models with support for parallel evaluation and result caching to reduce redundant API calls.

Aggregates evaluation results across problems and models with statistical analysis and report generation. Computes accuracy metrics, confidence intervals, error distributions, and comparative statistics; generates human-readable reports and machine-readable result files for further analysis. Supports filtering and slicing results by problem category, difficulty, or model for detailed performance analysis.

Implements domain-specific answer extraction from LLM outputs using pattern matching, parsing, and semantic analysis tailored to each problem type. For math problems, extracts numerical answers from LaTeX, symbolic notation, and natural language; for logic problems, validates logical conclusions; for code problems, extracts and validates generated code. Handles malformed outputs gracefully with detailed error reporting.

Classifies evaluation failures into specific error categories (syntax errors, runtime errors, logic errors, timeout, invalid format) with detailed error messages and logs. Provides aggregated error statistics showing which error types are most common across models and problems, enabling targeted debugging and model improvement. Supports custom error classification rules for domain-specific failure modes.

Ensures reproducible evaluation through dataset versioning, problem ID tracking, and result logging with full evaluation configuration capture. Stores evaluation metadata (model version, inference parameters, timestamp, dataset version) alongside results to enable exact reproduction of past evaluations. Supports result comparison across evaluation runs to track model improvements over time.

Orchestrates batch evaluation of multiple models across multiple datasets with configurable parallelization (thread/process-based) and resource management (rate limiting, memory constraints, timeout handling). The framework distributes evaluation tasks across available resources, monitors execution, handles failures gracefully with retry logic, and provides progress tracking and resource utilization metrics.