xCodeEval

Provides a standardized evaluation framework for code generation models that accepts generated code in 17 programming languages (C, C++, C#, Java, Kotlin, Go, Rust, Python, Ruby, PHP, JavaScript, Perl, Haskell, OCaml, Scala, D, Pascal) and validates correctness through actual execution against unit tests via the ExecEval Docker-based execution engine. Uses a centralized problem definition model with src_uid foreign keys linking generated code to shared problem descriptions and unittest_db.json, enabling consistent evaluation across language variants of the same problem.

Implements a foreign key linking system where all task-specific datasets (program synthesis, code translation, APR, retrieval) reference shared problem definitions via src_uid identifiers. Problem descriptions and unit tests are stored once in centralized problem_descriptions.jsonl and unittest_db.json files, then linked by src_uid to avoid duplication. The Hugging Face datasets API automatically resolves these links during data loading, returning enriched DatasetDict objects with problem context pre-joined to task examples.

Provides a Python API for loading xCodeEval datasets from Hugging Face Hub (NTU-NLP-sg/xCodeEval) with automatic src_uid-based linking between task datasets and shared problem definitions. The datasets library handles data downloading, caching, and streaming, while the xCodeEval integration automatically joins task examples with problem_descriptions.jsonl and unittest_db.json using src_uid foreign keys. Returns DatasetDict objects with enriched examples ready for model training or evaluation.

Provides an alternative data access method using Git LFS for users who prefer direct file access or need selective dataset downloads. Supports cloning the repository with LFS disabled, then pulling specific task files or problem definitions on demand. Useful for custom processing pipelines or environments where Python/Hugging Face is not available, though requires manual src_uid linking to join task examples with problem definitions.

Implements a standardized three-phase evaluation pipeline (Phase 1: Generation, Phase 2: Execution, Phase 3: Metrics) that applies consistently across all 7 tasks (program synthesis, code translation, APR, tag classification, code compilation, NL-code retrieval, code-code retrieval). Phase 1 generates or retrieves code, Phase 2 executes it via ExecEval or computes retrieval metrics, and Phase 3 aggregates results into pass@k, MRR, NDCG, or other task-specific metrics. Enables direct comparison of model performance across tasks.

Evaluates code generation models on the program synthesis task by accepting natural language problem descriptions and generating code solutions in any of 17 languages. The evaluation pipeline (Phase 1: Generation, Phase 2: Execution, Phase 3: Metrics) runs generated code against unit tests via ExecEval, computing pass@k metrics (pass@1, pass@10, etc.) that measure the probability of finding a correct solution within k samples. Supports both single-solution and multi-sample evaluation modes for assessing model reliability.

Evaluates code translation models by accepting source code in one language and generated translations in a target language, then validating functional equivalence through execution against shared unit tests. The translation evaluation pipeline compiles and executes both source and translated code against the same unittest_db.json test cases, comparing outputs to detect translation errors. Supports all 17 language pairs (though not all pairs may have training data) and uses language-specific compiler mappings to handle syntax differences.

Evaluates program repair models by providing buggy code snippets and expecting corrected versions that pass unit tests. The APR evaluation pipeline executes repaired code against unittest_db.json test cases, measuring whether the repair successfully fixes the bug without introducing new failures. Supports repairs across all 17 languages and uses the same execution-based validation as program synthesis, enabling direct comparison of repair quality.

Provides a retrieval task where models must find the correct code implementation given a natural language problem description, using a corpus of 7,500 unique code solutions across 17 languages. The retrieval evaluation uses semantic matching against a retrieval corpus (stored separately from task datasets) to measure ranking quality via metrics like MRR (Mean Reciprocal Rank) or NDCG. Supports both single-language and cross-language retrieval scenarios.

Evaluates code retrieval models on finding semantically similar code implementations given a query code snippet, using structural and semantic matching against the retrieval corpus. Unlike NL-code retrieval, this task measures code-to-code similarity across language variants of the same problem or functionally equivalent solutions in different languages. Supports both same-language and cross-language code matching.

Provides a code understanding task where models classify code snippets with semantic tags (e.g., algorithm type, data structure, complexity class). The tag classification dataset includes code examples with associated tags across all 17 languages, enabling evaluation of whether models understand code semantics beyond syntax. Uses standard multi-label classification metrics to measure tagging accuracy.

Provides a code compilation task that validates whether generated or translated code compiles successfully in its target language, using language-specific compiler mappings and configurations. The compilation evaluation is integrated into the ExecEval execution engine, which handles compiler invocation, error capture, and timeout management for each of the 17 supported languages. Returns detailed compilation errors and warnings for debugging.

Provides a containerized execution environment (ExecEval) that safely runs generated code in isolated Docker containers, with language-specific compiler and runtime configurations. The engine handles compilation, execution, timeout management, and output capture for all 17 languages, returning structured execution outcomes (pass/fail/timeout/error). Supports configurable timeout thresholds per language and prevents resource exhaustion through container limits.