AlpacaEval

Compares outputs from two models on identical instructions using an LLM (GPT-4, Claude, etc.) as an automatic judge. The PairwiseAnnotator class orchestrates three workflows: annotate_pairs() for pre-defined pairs, annotate_head2head() for full model-vs-model comparison, and annotate_samples() for random pair sampling. Supports pluggable decoder backends (OpenAI, Anthropic, Hugging Face, vLLM) with unified schema-based function calling to extract structured win/loss/tie judgments from judge LLM outputs.

Computes win rates between model pairs while controlling for output length bias through a length-aware normalization scheme. The system bins outputs by length percentile and calculates win rates within each bin, then aggregates to produce a length-controlled metric that prevents longer outputs from automatically winning. Implemented via processors that normalize comparison results before metric aggregation, addressing a core confound in LLM evaluation where verbosity correlates with perceived quality independent of actual instruction-following ability.

Integrates with Ollama, a lightweight model serving tool that simplifies running open-source LLMs locally. Users can run `ollama pull llama2` to download a model and `ollama serve` to start a local server, then point AlpacaEval to the Ollama endpoint. The integration handles HTTP requests to the Ollama API, supports streaming responses, and manages model lifecycle. Ollama is simpler to set up than vLLM and requires less GPU memory due to quantization, making it accessible to researchers without extensive infrastructure.

Ensures reproducible evaluation results by implementing deterministic sampling and random seeding throughout the pipeline. When sampling pairs from a large evaluation set, the system uses a fixed random seed to ensure the same pairs are selected across runs. Evaluation results are cached and reused if the same pairs are evaluated again. Configuration files include seed parameters that users can specify to control randomness. This enables researchers to share evaluation configurations and reproduce results exactly, critical for scientific rigor and benchmarking credibility.

Provides a unified abstraction layer for interacting with LLMs across multiple providers (OpenAI, Anthropic, Hugging Face, vLLM, Ollama) through a Decoder Registry pattern. Each provider has a concrete decoder implementation that handles authentication, API calls, response parsing, and caching. The system uses YAML-based model configurations to specify model names, API endpoints, and provider-specific parameters, allowing users to swap judge models or evaluation models without code changes. Supports both API-based (OpenAI, Anthropic) and self-hosted (vLLM, Ollama) deployments.

Extracts structured judgments (win/loss/tie) from judge LLM outputs using schema-based function calling and completion parsers. The system defines a schema for the judge's response (e.g., 'winner' field with enum values), sends it to the LLM via provider-specific function-calling APIs (OpenAI's tools, Anthropic's tool_use), and parses the structured response. Includes fallback completion parsers that extract judgments from free-form text if function calling fails, using regex and heuristic matching. This dual-path approach ensures robust judgment extraction even when LLMs don't strictly follow function-calling schemas.

Orchestrates large-scale evaluation runs by batching model outputs, managing API calls to judge models, caching results to avoid redundant evaluations, and aggregating judgments into final metrics. The main.py CLI entry point coordinates the workflow: loads model outputs and reference data, invokes the annotator system in batches, caches results per pair, and computes length-controlled win rates. Supports resumable evaluations where cached results are reused if re-running the same comparison, reducing cost and latency. Results are aggregated into leaderboard rankings with per-model statistics.

Generates ranked leaderboards from pairwise comparison results by aggregating win rates across all pairs and computing per-model statistics. The system calculates each model's win rate (wins / total comparisons), confidence intervals using binomial proportion methods, and sorts models by win rate. Supports filtering by instruction category, length range, or other metadata. Results are exported to CSV, JSON, or HTML formats for sharing and visualization. The leaderboard system handles ties and partial comparisons (where not all model pairs are evaluated).

Evaluates models on their ability to follow instructions by comparing outputs against reference outputs (e.g., human-written or expert-generated responses). The system loads instruction-output pairs, optionally includes reference outputs for context, and uses a judge LLM to assess whether each model output correctly follows the instruction. The judge considers factors like instruction adherence, completeness, and correctness. This differs from pairwise comparison by evaluating absolute quality against a reference rather than relative quality between two outputs.

Provides a command-line interface (main.py) that drives the entire evaluation pipeline through configuration files and command-line arguments. Users specify model outputs, judge model, evaluation parameters (batch size, number of samples, etc.), and output paths via YAML configs or CLI flags. The CLI orchestrates loading data, running the annotator, computing metrics, and generating leaderboards. Supports multiple evaluation modes (pairwise, head-to-head, sampling) and allows users to customize evaluation behavior without writing code. Configuration system uses YAML for model definitions and evaluation parameters.

Integrates with Hugging Face Hub to load and run models as judges or evaluation models. The system uses the Hugging Face transformers library to load models from the Hub, supports both local and remote model loading, and handles tokenization and inference. Users can specify any Hugging Face model as the judge by providing the model ID (e.g., 'meta-llama/Llama-2-7b-hf'). The integration supports both CPU and GPU inference, with automatic device management. This enables free or low-cost evaluation using open-source models instead of expensive API-based judges.

Integrates with vLLM (a high-performance LLM serving engine) to run judge models with optimized inference throughput and latency. The system connects to a vLLM server via HTTP API, sends batched requests, and receives structured responses. vLLM provides features like continuous batching, KV-cache optimization, and quantization that significantly speed up inference compared to naive transformers-based loading. Users can deploy a vLLM server locally or on a cluster, then point AlpacaEval to it via configuration. This enables fast, cost-effective evaluation at scale.