Multi Step Clinical Reasoning Evaluation Across Usmle Exam Hierarchy

via “multi-step clinical reasoning evaluation across usmle exam hierarchy”

12.7K USMLE medical exam questions for clinical AI evaluation.

Unique: Explicitly structures questions by USMLE step level and medical specialty, enabling evaluation of how LLMs progress through the licensing exam hierarchy and where reasoning breaks down; most medical QA datasets treat all questions equally without step-level stratification

vs others: Allows researchers to measure clinical reasoning progression across exam difficulty levels, whereas generic medical QA datasets (e.g., MedQA without step stratification) provide only aggregate accuracy and miss the critical insight of where models fail in the licensing pathway