MMLU (Massive Multitask Language Understanding)
BenchmarkFree57-subject benchmark, the standard metric for comparing LLMs.
Capabilities6 decomposed
multi-subject knowledge evaluation across 57 academic domains
Medium confidenceEvaluates LLM knowledge breadth and depth across 57 distinct academic subjects (mathematics, physics, chemistry, biology, history, law, medicine, engineering, philosophy, etc.) using 15,908 curated multiple-choice questions. The dataset stratifies questions by difficulty level from elementary to professional certification exams, enabling fine-grained assessment of model performance across knowledge domains and cognitive complexity tiers. Scoring is deterministic (exact match on selected choice) and comparable across models.
Combines breadth (57 subjects) with depth (difficulty stratification from elementary to professional certification level) in a single unified benchmark, with 15,908 questions curated from real academic and professional exams rather than synthetic generation. The subject taxonomy spans STEM, humanities, and professional domains in a way that no single-domain benchmark achieves.
More comprehensive and domain-balanced than HellaSwag (entertainment focus) or ARC (science-only), and more standardized than ad-hoc evaluation sets because it's widely adopted as the de facto metric for comparing frontier LLMs in published research.
difficulty-stratified performance analysis
Medium confidenceSegments the 15,908 questions into difficulty tiers (elementary, high school, college, professional) enabling builders to measure whether a model's knowledge is shallow pattern-matching or deep understanding. Each question is tagged with difficulty metadata, allowing disaggregated scoring that reveals performance cliffs — e.g., a model may score 85% on high school questions but only 40% on professional-level law or medicine questions. This stratification exposes whether improvements are broad-based or concentrated in easier domains.
Explicitly tags questions with difficulty levels derived from real academic curricula (elementary through professional certification), enabling builders to measure reasoning depth rather than just aggregate knowledge. Most benchmarks report a single score; MMLU's stratification reveals whether improvements are broad or concentrated in easy questions.
Provides finer-grained difficulty analysis than GSM8K (math-only) or TruthfulQA (single-domain), and the difficulty labels are grounded in real educational standards rather than arbitrary heuristics.
subject-specific knowledge profiling
Medium confidenceOrganizes 15,908 questions into 57 distinct subject categories (mathematics, physics, chemistry, biology, history, law, medicine, engineering, philosophy, economics, etc.), enabling builders to generate per-subject accuracy profiles. Each question is tagged with its subject, allowing disaggregated scoring that reveals domain-specific strengths and weaknesses. A model might score 90% on STEM subjects but only 60% on humanities, or vice versa. This enables targeted evaluation for domain-specific applications.
Covers 57 distinct subjects spanning STEM, humanities, social sciences, and professional domains in a single benchmark, providing comprehensive domain coverage that no single-subject benchmark achieves. Subject taxonomy is derived from real academic curricula and professional certification exams.
Broader subject coverage than domain-specific benchmarks (e.g., MedQA for medicine only) while maintaining standardization across all subjects, enabling both broad knowledge assessment and targeted domain evaluation in one dataset.
standardized model comparison and ranking
Medium confidenceProvides a canonical, widely-adopted benchmark for comparing LLM capabilities across the industry. MMLU is the single most reported metric in LLM research papers and model cards, enabling builders to position their models against published baselines (GPT-4, Claude, Llama, etc.). Scoring is deterministic and reproducible: exact match on multiple-choice selection. The dataset is fixed and versioned, ensuring that comparisons across papers and time periods are valid. Leaderboards and published results enable quick competitive analysis.
De facto industry standard for LLM evaluation, with results published in virtually every major LLM research paper and model card since 2021. Canonical dataset version ensures reproducibility across papers and time periods, unlike ad-hoc evaluation sets that vary between researchers.
More widely adopted and cited than competing benchmarks (ARC, HellaSwag, TruthfulQA), making it the single most reliable metric for comparing published LLM capabilities and positioning new models in the competitive landscape.
reproducible evaluation with fixed question set
Medium confidenceProvides a fixed, versioned dataset of 15,908 questions that doesn't change between evaluation runs, enabling reproducible and comparable results across different models, teams, and time periods. The dataset is immutable and publicly available on Hugging Face, ensuring that any builder can download the exact same questions and verify published results. This eliminates variance from question generation, sampling, or dataset drift that would occur with dynamic benchmarks.
Immutable, versioned dataset published on Hugging Face ensures that any builder can download and evaluate against the exact same 15,908 questions used in published research. No question generation variance, sampling randomness, or dataset drift between evaluation runs.
More reproducible than dynamically-generated benchmarks or evaluation sets that vary between researchers; enables verification of published results and fair comparison across models and time periods.
professional certification exam alignment
Medium confidenceIncludes questions sourced from or aligned with real professional certification exams (law bar exams, medical licensing exams, engineering professional exams, etc.), enabling evaluation of whether LLMs can perform at professional-grade levels. Questions are tagged with difficulty levels that correspond to actual exam difficulty, and some questions are directly sourced from published exam materials. This grounds the benchmark in real-world professional standards rather than synthetic or academic-only questions.
Includes questions sourced from or aligned with real professional certification exams (law bar, medical licensing, engineering professional exams), grounding the benchmark in actual professional standards rather than purely academic questions. Professional-level questions are explicitly tagged and stratified.
More professionally-grounded than purely academic benchmarks (e.g., SQuAD, which focuses on reading comprehension) while maintaining breadth across multiple professional domains in a single dataset.
Capabilities are decomposed by AI analysis. Each maps to specific user intents and improves with match feedback.
Related Artifactssharing capabilities
Artifacts that share capabilities with MMLU (Massive Multitask Language Understanding), ranked by overlap. Discovered automatically through the match graph.
mmlu
Dataset by cais. 4,76,392 downloads.
MMLU
57-subject knowledge benchmark — 15K+ questions across STEM, humanities, professional domains.
MMMU
Expert-level multimodal understanding across 30 subjects.
MATH
12.5K competition math problems across 7 subjects and 5 difficulty levels.
ARC (AI2 Reasoning Challenge)
7.8K science questions testing genuine reasoning, not just recall.
Atlas
Revolutionizes studying with tailored, AI-driven academic...
Best For
- ✓ML researchers and model developers benchmarking LLM capabilities
- ✓Organizations evaluating commercial LLMs for knowledge-intensive applications
- ✓Teams building domain-specific LLMs who need standardized evaluation
- ✓Model developers optimizing for professional-grade applications
- ✓Teams evaluating whether an LLM is production-ready for high-stakes domains
- ✓Researchers studying scaling laws and whether model size correlates with reasoning depth
- ✓Organizations building domain-specific LLMs (medical, legal, financial) who need targeted evaluation
- ✓Researchers studying how training data composition affects knowledge distribution
Known Limitations
- ⚠Multiple-choice format doesn't measure reasoning depth or ability to generate novel solutions — only recognition and selection
- ⚠No evaluation of explanation quality or reasoning chains; a model can guess correctly without understanding
- ⚠Subject distribution is imbalanced (e.g., more STEM than humanities questions), skewing aggregate scores
- ⚠Static snapshot of knowledge as of dataset creation date; doesn't measure ability to learn or update knowledge
- ⚠English-only; no multilingual evaluation despite many LLMs supporting 100+ languages
- ⚠Difficulty labels are subjective and assigned by dataset creators; no consensus on what 'professional' means across domains
Requirements
Input / Output
UnfragileRank
UnfragileRank is computed from adoption signals, documentation quality, ecosystem connectivity, match graph feedback, and freshness. No artifact can pay for a higher rank.
About
The standard benchmark for evaluating LLM knowledge and reasoning across 57 academic subjects spanning STEM, humanities, social sciences, and professional domains. 15,908 multiple-choice questions at difficulty levels from elementary to professional (law, medicine, engineering). Originally by Hendrycks et al., now the single most reported metric for comparing language models. Tests knowledge breadth and reasoning depth. Scores range from 25% (random) to 90%+ for frontier models.
Categories
Alternatives to MMLU (Massive Multitask Language Understanding)
Open-source image generation — SD3, SDXL, massive ecosystem of LoRAs, ControlNets, runs locally.
Compare →Are you the builder of MMLU (Massive Multitask Language Understanding)?
Claim this artifact to get a verified badge, access match analytics, see which intents users search for, and manage your listing.
Get the weekly brief
New tools, rising stars, and what's actually worth your time. No spam.
Data Sources
Looking for something else?
Search →