Reproducible Evaluation With Version Control And Result Archiving

via “benchmark reproducibility and versioning”

AI coding agent benchmark — real GitHub issues, end-to-end evaluation, the standard for code agents.

Unique: Pins all 12 repositories to specific commits and includes dependency lock files, ensuring that benchmark instances are identical across runs and time periods. This is critical for academic research where reproducibility is essential and for tracking long-term progress where code changes would confound results.

vs others: More reproducible than live benchmarks that pull from current repository state because fixed commits prevent code changes from invalidating previous results, and more practical than manual snapshot management because versioning is automated and documented.