Benchmark Version Management And Reproducibility

via “benchmark reproducibility and versioning”

AI coding agent benchmark — real GitHub issues, end-to-end evaluation, the standard for code agents.

Unique: Pins all 12 repositories to specific commits and includes dependency lock files, ensuring that benchmark instances are identical across runs and time periods. This is critical for academic research where reproducibility is essential and for tracking long-term progress where code changes would confound results.

vs others: More reproducible than live benchmarks that pull from current repository state because fixed commits prevent code changes from invalidating previous results, and more practical than manual snapshot management because versioning is automated and documented.

via “benchmark reproducibility and versioning”

Zero-shot LLM evaluation for reasoning tasks.

Unique: Captures full evaluation provenance (model version, inference parameters, dataset version, timestamp) alongside results, enabling exact reproduction and comparison of evaluations across time

vs others: More rigorous than ad-hoc evaluation; systematic versioning and metadata capture enable transparent, reproducible benchmarking suitable for publication and long-term tracking

via “benchmark reproducibility through fixed question sets and seed management”

Multi-turn conversation benchmark — 80 questions, 8 categories, GPT-4 as judge.

Unique: Treats reproducibility as a first-class concern by versioning questions, recording all inference parameters, and publishing metadata alongside results. Questions are public, enabling external verification.

vs others: More reproducible than proprietary benchmarks (which don't publish questions); more rigorous than informal evaluation practices that don't track parameters.

via “benchmark versioning and continuous improvement”

Real OS benchmark for multimodal computer agents.

Unique: Actively maintains and improves benchmark with documented versions and community-driven bug fixes, rather than releasing a static benchmark. The 2025-07-28 'OSWorld-Verified' update indicates responsiveness to community feedback and ongoing refinement.

vs others: More maintainable and trustworthy than static benchmarks because improvements are tracked and documented, but requires users to specify version for reproducibility and may introduce incompatibilities between versions.

via “benchmark dataset versioning and curation pipeline”

Benchmark for dangerous knowledge in LLMs.

Unique: Implements a formal curation pipeline with expert validation and inter-rater agreement checks, rather than ad-hoc question collection. Versioning enables reproducible research and transparent tracking of benchmark evolution.

vs others: More rigorous than informal benchmarks because it enforces expert review, inter-rater validation, and version control, reducing bias and enabling reproducible comparisons across papers.

via “snapshot versioning and baseline management with rollback capability”

Visual testing platform with AI-powered regression detection.

Unique: Maintains complete version history of visual baselines linked to commits/PRs, enabling rollback and historical comparison. Percy automatically manages baseline branching for feature branches, eliminating manual baseline synchronization.

vs others: More sophisticated than BackstopJS's file-based baseline management (which requires manual Git tracking) and provides better audit trails than Chromatic's implicit baseline versioning; enables compliance-grade visual change tracking.

via “benchmark-version-management-and-reproducibility”

open_llm_leaderboard — AI demo on HuggingFace

Unique: Maintains explicit version pinning for benchmark datasets and evaluation code, enabling researchers to reproduce exact evaluation conditions and compare models across leaderboard updates with different benchmark versions

vs others: More reproducible than leaderboards with floating benchmark versions (enables exact reproduction) and more transparent than closed benchmarking services (version history is documented and accessible)

via “dataset versioning and reproducible snapshot loading”

Dataset by lavita. 5,55,826 downloads.

Unique: Leverages HuggingFace Hub's Git-based versioning infrastructure to provide immutable dataset snapshots with full history tracking. Enables citation-grade reproducibility through semantic versioning and automatic version pinning in code.

vs others: More reproducible than ad-hoc dataset downloads because versions are immutable and citable; better than manual versioning because Git history is automatically maintained and queryable

via “version-control-and-reproducibility”

Dataset by huggingface. 25,31,937 downloads.

Unique: Leverages HuggingFace's git-based versioning infrastructure to provide dataset version control as a first-class feature, eliminating the need for manual snapshot management or external version control systems

vs others: More integrated than external version control (DVC, Pachyderm) because versioning is built into the dataset platform itself, and more transparent than snapshot-based systems because full git history is queryable

via “benchmark dataset versioning and provenance tracking”

Dataset by xlangai. 11,02,516 downloads.

Unique: Tracks dataset version, OSWorld benchmark version, Ubuntu system configuration, and execution environment metadata for each cached trajectory, enabling reproducible evaluation and transparent tracking of benchmark evolution

vs others: Provides explicit provenance tracking for OS task datasets, enabling reproducibility and version-aware evaluation that alternatives lacking metadata context cannot support

via “test data versioning and reproducibility”

via “model-versioning-and-management”

via “model versioning and deployment management”

via “manage-model-versions-and-history”

via “model versioning and experiment tracking”