Big Code Bench

Evaluates LLM code generation across 1,140 realistic programming tasks organized into two splits (Complete for all models, Instruct for chat models) using pass@k statistical metrics that measure the probability at least one of k generated samples passes all test cases. The system generates multiple code samples per task, executes each against embedded test suites, and aggregates results into pass@1, pass@10, pass@100 metrics for comparative model analysis.

Provides a unified Python API that abstracts away provider-specific differences (OpenAI, Anthropic, Hugging Face, Ollama, vLLM) through a standardized code generation interface. The system handles provider-specific authentication, API formatting, parameter mapping, and response parsing, allowing users to swap models without changing benchmark code. Internally routes requests through provider-specific adapters that normalize temperature, max_tokens, and sampling parameters.

Executes generated Python code in isolated environments using three configurable backends: local execution with resource limits, E2B sandbox for remote secure execution, and Hugging Face Gradio spaces for zero-setup remote evaluation. Each backend enforces execution timeouts, memory limits, and exception handling to prevent malicious or infinite-loop code from crashing the evaluation system. Results include execution status, stdout/stderr capture, and test case pass/fail verdicts.

Pre-processes generated code through syntax checking (via ast.parse) and sanitization to remove unsafe patterns before execution. The syncheck command validates Python syntax without executing, catching parse errors early. Sanitization removes or neutralizes dangerous constructs (eval, exec, __import__, file operations) while preserving functional code. This two-stage filtering reduces execution errors and improves test reliability by ensuring only valid, safe code reaches the sandbox.

Manages 1,140 code generation tasks organized into two splits (Complete: docstring-based for all models, Instruct: natural language for chat models) and two subsets (full: all 1,140 tasks, hard: 148 challenging tasks). Each task includes function signature, docstring/instruction, test cases, and metadata. The system loads tasks from JSONL files, filters by split/subset, and provides task iteration for batch evaluation. Metadata includes task difficulty, required libraries, and test case counts.

Aggregates per-task evaluation results into pass@k metrics (pass@1, pass@10, pass@100) that measure the probability at least one of k samples passes all test cases. Implements statistical calculation: pass@k = 1 - C(n-c, k) / C(n, k) where n is total samples and c is passing samples. Stores results in structured JSON format with per-task verdicts, sample-level details, and aggregate metrics. The inspect command provides detailed result analysis and leaderboard-compatible output.

Generates multiple code samples per task with configurable temperature and sampling parameters (top_p, top_k, frequency_penalty) to explore model output diversity. The run_codegen() function orchestrates batch generation across all tasks, managing API calls, rate limiting, and result persistence. Supports generating n_samples (typically 1, 10, 100) per task with different random seeds to ensure diversity. Results are stored in JSONL format with model name, task ID, sample index, and generated code.

Provides Docker container templates (e2b.Dockerfile, e2b.toml) for creating reproducible evaluation environments with pinned Python versions, library versions, and system dependencies. Containers include pre-installed libraries (NumPy, Pandas, Matplotlib, etc.) required by benchmark tasks. E2B integration enables remote execution of containers with automatic cleanup and resource isolation. This ensures evaluation results are reproducible across different machines and time periods.

The inspect command provides comprehensive analysis of evaluation results including per-task pass/fail verdicts, sample-level details, error categorization, and performance statistics. Generates human-readable reports showing which tasks passed, which failed, and why (syntax error, timeout, test failure, exception). Supports filtering by task category, difficulty, and library to identify model weaknesses. Results can be exported in multiple formats (JSON, CSV, markdown) for further analysis.

Formats evaluation results in standardized JSON schema compatible with public leaderboards (e.g., Hugging Face model hub). Results include model metadata (name, version, provider), evaluation metadata (date, split, subset, n_samples), and per-task results with pass@k metrics. The system generates leaderboard-ready files that can be directly submitted to benchmarking platforms without manual reformatting. Supports versioning and result comparison across model iterations.