Multi Hop Reasoning Evaluation Across Document Sections

via “multi-hop reasoning evaluation across document sections”

8.3K financial reasoning questions over real S&P 500 earnings reports.

Unique: Embeds multi-hop reasoning requirements within authentic financial documents where hops correspond to real relationships between financial statement sections, rather than synthetic reasoning chains. This tests whether models understand domain structure, not just generic multi-hop patterns.

vs others: More realistic than synthetic multi-hop datasets (HotpotQA, 2WikiMultiHopQA) because reasoning hops follow actual financial relationships, but less controlled because document structure varies and reasoning paths are implicit rather than explicitly annotated

via “compositional reasoning benchmark with multi-document retrieval requirements”

113K questions requiring multi-hop reasoning across Wikipedia articles.

Unique: Explicitly validates that questions require multi-hop reasoning through crowdsourced verification that single-document retrieval cannot answer them. Questions are structured around entity linking and relationship composition, forcing systems to perform genuine multi-stage reasoning rather than single-stage retrieval.

vs others: Compared to general QA datasets like Natural Questions (single-hop, web-scale) or SQuAD (single-document), HotpotQA's explicit multi-hop requirement and supporting fact annotations make it uniquely suited for evaluating whether systems perform compositional reasoning vs. pattern matching.

via “multi-hop-document-reasoning”

An open-source platform for building and evaluating RAG and agentic applications. [#opensource](https://github.com/agentset-ai/agentset)

Unique: Implements iterative retrieval-augmented reasoning where the LLM generates follow-up queries based on retrieved context, rather than executing a fixed retrieval plan. This allows dynamic exploration of document relationships without pre-computed knowledge graphs.

vs others: Simpler than graph-based RAG (no knowledge graph construction required) but more flexible than single-hop retrieval; faster than manual multi-document analysis because retrieval and synthesis are automated.