Diagnostic Accuracy Benchmarking And Quality Assurance

via “multi-annotator agreement and answer quality assessment”

307K real Google Search queries answered from Wikipedia.

Unique: Includes explicit inter-annotator agreement metrics for each question, enabling researchers to understand benchmark reliability and filter by agreement level

vs others: More transparent about annotation quality than benchmarks that hide disagreement, allowing researchers to make informed decisions about evaluation methodology