Human Baseline Performance Anchoring

via “human-baseline performance anchoring”

23 hardest BIG-Bench tasks where models initially failed.

Unique: Explicitly selected tasks where models underperformed humans at time of curation, creating a self-calibrated hard benchmark where human performance is the reference point rather than an afterthought. This selection strategy ensures the benchmark remains challenging as models improve.

vs others: More rigorous than benchmarks without human baselines because it enables quantitative model-vs-human comparison; more meaningful than benchmarks where humans outperform models by large margins, which may indicate task misalignment rather than genuine reasoning difficulty.

via “human-performance-anchored difficulty calibration”

44K pronoun resolution problems testing commonsense understanding.

Unique: Establishes 94% human performance as an explicit calibration anchor through expert annotation, enabling quantitative model-human comparison rather than abstract performance claims; this anchor is embedded in dataset metadata and evaluation harnesses

vs others: More interpretable than relative benchmarks (e.g., 'better than GPT-3') because human performance provides an absolute reference point; more rigorous than datasets without human baselines where model performance claims lack grounding