HotpotQA

Provides 113,000 question-answer pairs where each question requires traversing and reasoning across 2+ Wikipedia articles to derive the answer. The dataset includes explicit supporting fact annotations identifying which sentences from source documents are necessary for answering, enabling training of models that can both answer questions and explain their reasoning chains. Built through crowdsourced annotation with quality control mechanisms to ensure multi-hop reasoning is genuinely required rather than answerable from single documents.

Provides a structured evaluation methodology for assessing whether QA systems can correctly identify which source sentences support their answers. The framework compares predicted supporting facts against human-annotated ground truth using precision, recall, and F1 metrics at both sentence and paragraph levels. This enables measurement of reasoning transparency independent of answer correctness, allowing diagnosis of whether a system found the right answer for the right reasons.

Structures questions to require explicit composition of facts across multiple Wikipedia articles, creating a benchmark where naive single-document retrieval fails. Questions are designed such that the answer cannot be found in any single article; instead, the system must retrieve multiple relevant documents, identify the connecting entity or relationship, and synthesize information across them. This tests whether systems can perform true multi-hop reasoning versus pattern matching on single documents.

Provides a controlled evaluation setting where systems must distinguish relevant documents from distractors. The dataset includes both supporting documents (necessary for answering) and distractor documents (related to the question but not required for the answer). This tests whether retrieval systems can rank supporting documents above distractors, a critical capability for multi-hop QA where false positives in retrieval compound through reasoning stages. Evaluation measures whether systems retrieve all necessary documents while minimizing false positives.

Categorizes questions into distinct reasoning types (e.g., 'bridge' questions requiring entity linking between documents, 'comparison' questions requiring fact synthesis) and provides labels enabling analysis of system performance across reasoning patterns. This allows fine-grained evaluation of which reasoning types systems handle well vs. poorly, and enables targeted training or evaluation on specific compositional reasoning challenges. The taxonomy captures the structural reasoning requirements independent of domain content.

Questions are generated from Wikipedia articles and require reasoning over real-world entities, relationships, and facts. This grounds reasoning in a concrete knowledge domain (Wikipedia) rather than synthetic or template-based questions, enabling evaluation of whether systems can handle real-world complexity. Questions span diverse topics (people, places, films, organizations) and reasoning patterns (attribute lookup, entity linking, relationship chaining).