AlpacaEval

Automatically evaluates instruction-following model outputs by using a judge LLM (GPT-4, Claude, etc.) to perform pairwise comparisons between two model responses on the same instruction. Implements length-controlled win rate calculation that normalizes for output length bias by penalizing verbosity, preventing longer but lower-quality outputs from unfairly winning comparisons. The system uses configurable judge prompts and completion parsers to extract structured win/loss decisions from judge LLM outputs.

Abstracts interactions with different LLM providers (OpenAI, Anthropic, Hugging Face, vLLM) through a unified Decoder interface and registry system. Each provider has a dedicated decoder class that handles authentication, API calls, response parsing, and caching. The system supports both API-based models (GPT-4, Claude) and local inference engines (vLLM, Ollama), with automatic fallback and retry logic for failed requests.

Validates and preprocesses model outputs before evaluation, including format checking (JSON structure), field validation (required 'instruction' and 'output' fields), and optional cleaning (whitespace normalization, encoding fixes). Detects and reports malformed outputs that would cause evaluation to fail. Supports multiple input formats (JSON, JSONL, CSV) with automatic format detection and conversion to internal representation.

Enables reproducible evaluations by capturing all evaluation parameters (judge model, prompt template, length penalty, random seed) in YAML configuration files that can be version-controlled and shared. Evaluation results include metadata (configuration hash, evaluation date, judge model version) allowing tracing back to exact evaluation setup. Supports loading prior configurations to reproduce historical evaluation runs.

Allows customization of the prompt template used to instruct the judge LLM on how to compare two model outputs. Supports multiple evaluation methodologies (pairwise comparison, ranking, scoring) through different prompt templates stored as YAML configurations. Includes a completion parser system that extracts structured decisions (win/loss/tie) from free-form judge LLM outputs using regex patterns and heuristics, handling cases where the judge outputs ambiguous or malformed responses.

Orchestrates evaluation of multiple model pairs through three modes: (1) annotate_pairs() for evaluating pre-specified pairs, (2) annotate_head2head() for comparing two models across all instructions, and (3) annotate_samples() for randomly sampling pairs from a larger set of models. Implements efficient batching of judge requests to reduce API calls, with optional parallel execution across multiple judge instances. Supports tournament-style evaluation where models are ranked through transitive comparisons.

Computes a length-adjusted win rate that penalizes longer outputs to control for length bias. The metric applies a configurable length penalty function (e.g., exponential decay) to the raw win rate based on the difference in output lengths between the two models being compared. Implemented in the metrics calculation pipeline, this allows fair comparison between verbose and concise models by normalizing for the confound that judges tend to prefer longer responses.

Aggregates pairwise comparison results into ranked leaderboards showing each model's win rate, number of comparisons, and ranking position. Supports multiple export formats (CSV, JSON, HTML) and includes statistical summaries (mean win rate, standard deviation, confidence intervals). The leaderboard system handles ties and incomplete comparisons, and can generate both overall rankings and per-category breakdowns (e.g., by instruction type or difficulty).

Provides command-line interface for running complete evaluation workflows from model outputs to leaderboard generation. The CLI accepts configuration files (YAML) specifying model paths, judge settings, evaluation mode, and output options. Implements a main.py entry point that orchestrates the full pipeline: loading model outputs, running pairwise comparisons, calculating metrics, and exporting results. Supports both interactive and batch modes for integration into CI/CD workflows.

Provides a curated dataset of 805 instruction-following examples designed to evaluate general-purpose LLM instruction-following ability. The dataset is included with the package and can be loaded programmatically or via CLI. Includes instructions across diverse categories (writing, math, coding, reasoning) with varying difficulty levels. Supports custom instruction datasets by accepting JSON/JSONL files with 'instruction' and optional 'reference_output' fields.

Implements a file-based cache that stores judge LLM responses to avoid re-evaluating identical instruction pairs. The cache uses instruction and model output hashes as keys, enabling deduplication across multiple evaluation runs. When a cached result is found, the system returns the cached judgment without calling the judge LLM, reducing API costs and latency. Cache can be cleared or inspected via CLI commands.

Implements exponential backoff retry logic for failed judge API calls, with configurable retry counts and backoff parameters. Handles common failure modes: rate limiting (429), temporary service unavailability (5xx), and network timeouts. Failed requests are logged with context (instruction, models, error details) for debugging. Supports graceful degradation where partial evaluation results are returned if some comparisons fail.