Benchmark Competitive Performance Across Reasoning Tasks

via “multi-domain reasoning task stratification”

23 hardest BIG-Bench tasks where models initially failed.

Unique: Explicitly stratifies tasks by reasoning modality (algorithmic, arithmetic, logical, causal, spatial) rather than treating all hard tasks as monolithic, enabling domain-specific capability assessment. This structure allows researchers to correlate model architecture choices with specific reasoning strengths.

vs others: More analytically useful than generic hard task collections because stratification enables root-cause analysis of reasoning failures; more focused than full BIG-Bench which lacks explicit domain organization.

via “reasoning and multi-step problem solving”

Microsoft's 3.8B model with 128K context for edge deployment.

Unique: Achieves 69% MMLU reasoning performance in a 3.8B model through synthetic training data specifically designed for reasoning patterns, significantly outperforming typical SLMs on reasoning benchmarks despite extreme parameter efficiency

vs others: Delivers reasoning capability in 3.8B parameters (vs. Mistral 7B, Llama 3.2 1B which don't emphasize reasoning) while remaining mobile-deployable, trading some accuracy for extreme efficiency and edge compatibility

via “state-of-the-art visual reasoning on open-weight benchmarks”

Meta's largest open multimodal model at 90B parameters.

Unique: Claims state-of-the-art performance specifically on open-weight benchmarks (not all benchmarks), positioning it as the strongest available open-source alternative rather than claiming parity with proprietary systems across all metrics

vs others: Larger parameter count (90B vs typical 34B open models) enables stronger reasoning, though actual benchmark scores remain undocumented and unverifiable from public sources

via “cross-model reasoning capability comparison”

7.8K science questions testing genuine reasoning, not just recall.

Unique: Provides a reasoning-specific evaluation surface (Challenge set curated to exclude shallow-method-solvable questions) that isolates reasoning capability from retrieval capability, enabling cleaner comparison of how different models approach reasoning tasks. Domain stratification further enables analysis of whether reasoning capability is uniform or domain-specific.

vs others: More suitable for reasoning-focused comparison than generic QA benchmarks because Challenge set explicitly filters out retrieval-solvable questions; more fine-grained than single-metric leaderboards because it supports domain and difficulty stratification

via “benchmark-validated reasoning performance on standardized datasets”

Alibaba's 32B reasoning model with chain-of-thought.

Unique: Provides documented benchmark results on standardized reasoning datasets (AIME 79.5%, MATH-500 96.4%) enabling quantitative performance validation, with explicit comparison claims against larger models

vs others: Demonstrates competitive reasoning performance on standardized benchmarks comparable to much larger models, providing quantitative evidence of reasoning capability for evaluation and comparison purposes

via “reasoning and chain-of-thought decomposition for complex tasks”

Google's open-weight model family from 1B to 27B parameters.

Unique: 27B variant achieves reasoning performance competitive with much larger models (70B+) through optimized training on reasoning-heavy datasets and learned chain-of-thought patterns, without requiring external reasoning engines or symbolic solvers

vs others: Outperforms Llama 2 70B on math and coding reasoning benchmarks while being 2.6x smaller, and matches Mistral 7B on reasoning tasks while offering superior code generation quality

via “reasoning and multi-step problem decomposition”

TII's 180B model trained on curated RefinedWeb data.

Unique: Achieves strong reasoning performance through scale (180B parameters) and data quality (3.5T meticulously-cleaned RefinedWeb tokens) rather than specialized reasoning fine-tuning, enabling emergent reasoning capabilities across diverse domains without task-specific training.

vs others: Larger parameter count than reasoning-specialized models like Llama 2 70B enables better few-shot reasoning, but lacks explicit chain-of-thought fine-tuning that models like GPT-4 or Claude employ, potentially requiring more sophisticated prompting to achieve comparable reasoning quality.

via “mathematical reasoning with math benchmark performance”

Meta's 70B open model matching 405B-class performance.

Unique: Achieves strong mathematical reasoning performance at 70B parameters through instruction-tuning on mathematical problem-solving datasets, enabling competitive MATH benchmark performance without specialized symbolic reasoning modules

vs others: Provides mathematical reasoning capability comparable to larger closed-source models while remaining open-weight and self-hostable, though without formal verification guarantees of symbolic math systems

via “multi-step mathematical reasoning benchmark evaluation”

8.5K grade school math problems — multi-step reasoning, verifiable solutions, reasoning benchmark.

Unique: Uses linguistically diverse, human-authored grade school problems (not synthetic) that require genuine multi-step reasoning with basic arithmetic, combined with a standardized answer extraction format (#### delimiter) that enables reproducible evaluation across heterogeneous model outputs

vs others: More challenging than simple arithmetic benchmarks (requires 2-8 reasoning steps) yet more accessible than advanced math benchmarks, making it ideal for measuring practical reasoning improvements in production models

via “benchmark-validated reasoning performance”

01.AI's high-performance reasoning model.

Unique: unknown — insufficient data on which benchmarks were used, evaluation methodology, and how performance compares to GPT-4, Claude 3, or Llama 3 on specific reasoning tasks

vs others: Claims top benchmark performance but provides no comparative data, making it impossible to assess whether Yi-Lightning outperforms or underperforms established models like GPT-4 or Claude on standard reasoning benchmarks

via “evaluation metric formulation”

Abstraction and reasoning corpus for general intelligence

Unique: The evaluation metrics are specifically tailored to assess abstract reasoning capabilities, unlike generic metrics that may not reflect reasoning depth.

vs others: Offers more nuanced evaluation than traditional benchmarks like accuracy, which may not fully capture reasoning abilities.

via “task-specific baseline comparison”

Subset of BIG-Bench where most models fail

Unique: Utilizes a curated set of benchmarks that focus on reasoning tasks, providing a more relevant comparison than general performance metrics.

vs others: Offers a more nuanced view of model performance by focusing specifically on reasoning-related tasks, unlike broader benchmarks.

via “reasoning-specialized model identification and separate ranking”

ReLE评测：中文AI大模型能力评测（持续更新）：目前已囊括374个大模型，覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型， 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜，也提供规模超200万的大

Unique: Identifies and separately ranks reasoning-specialized models (e.g., DeepSeek-R1, o1-mini) in dedicated leaderboard (reasonmodel.md) rather than mixing with general-purpose models. Recognizes that reasoning-specialized models have distinct performance profiles and enables category-specific comparison. Maintains separate ranking for models optimized for complex reasoning tasks.

vs others: Explicit reasoning-specialist categorization vs single global leaderboard (which obscures reasoning-specialization benefits) and dedicated reasoning evaluation vs general benchmarks

via “benchmark evaluation on multi-hop reasoning tasks”

[ICML 2024] LLMCompiler: An LLM Compiler for Parallel Function Calling

Unique: Provides built-in evaluation on standard multi-hop reasoning benchmarks (HotpotQA, ParallelQA) with metrics for accuracy, latency, and cost, enabling quantitative assessment of planning and execution efficiency.

vs others: More comprehensive than simple accuracy measurement because it includes latency and cost metrics; enables direct comparison of parallel vs sequential execution on standard benchmarks.

via “specialized capability indexing for coding and reasoning tasks”

Artificial Analysis provides objective benchmarks & information to help choose AI models and hosting providers.

Unique: Separates model evaluation by task domain (coding, reasoning, agentic) rather than treating all models as general-purpose, recognizing that a model's strength in one domain doesn't guarantee strength in another. The reasoning capability indicator provides a quick filter for models suitable for complex reasoning tasks.

vs others: More targeted than general leaderboards because it isolates performance on specific task types; more practical for specialists than one-size-fits-all rankings; more discoverable than searching individual benchmark papers because indices are pre-computed and filterable.

via “reasoning and step-by-step problem decomposition”

Gemma 4 26B A4B IT is an instruction-tuned Mixture-of-Experts (MoE) model from Google DeepMind. Despite 25.2B total parameters, only 3.8B activate per token during inference — delivering near-31B quality at...

Unique: MoE expert specialization enables dedicated reasoning experts that activate for complex reasoning tasks, while general-purpose experts handle simpler steps, optimizing compute allocation across reasoning complexity

vs others: Provides faster reasoning than Llama 3.1 8B (15-20% speedup) while maintaining comparable accuracy on grade-school math and logic puzzles, though underperforms specialized reasoning models like o1-mini on competition-level problems

via “reasoning and chain-of-thought problem solving”

Meta's Llama 3.1 — high-quality text generation and reasoning

Unique: Explicitly trained for chain-of-thought reasoning across all three variants, with the 405B model claiming state-of-the-art performance. Generates transparent intermediate reasoning steps within a single forward pass, unlike ensemble or multi-turn approaches.

vs others: Provides transparent reasoning comparable to Claude 3.5 Sonnet and GPT-4o, but runs locally without API calls. Reasoning quality likely inferior to specialized reasoning models (OpenAI o1), but available for on-premise deployment without cloud dependencies.

via “mathematical reasoning evaluation”

UGI-Leaderboard — AI demo on HuggingFace

Unique: Isolates mathematical reasoning as a distinct evaluation dimension on the leaderboard, enabling models to be ranked separately on math vs general generation, revealing capability specialization.

vs others: Simpler than running MATH or GSM8K locally with custom evaluation scripts, but less transparent than open-source math benchmarks regarding problem selection and difficulty.

via “complex reasoning and chain-of-thought decomposition”

Command R7B (12-2024) is a small, fast update of the Command R+ model, delivered in December 2024. It excels at RAG, tool use, agents, and similar tasks requiring complex reasoning...

Unique: Command R7B's reasoning is optimized for RAG and tool-use contexts, where intermediate steps can reference retrieved documents or tool outputs, enabling grounded reasoning that combines external knowledge with logical inference

vs others: Outperforms GPT-4 on MATH and AIME benchmarks when combined with tool use for calculation, because it can delegate computation to tools rather than attempting symbolic math in-context

via “semantic reasoning with chain-of-thought decomposition”

The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language...

Unique: Trained on reasoning-focused datasets to naturally emit intermediate reasoning tokens without explicit prompting, using transformer attention patterns that learn to decompose problems into sub-steps, enabling transparent multi-hop reasoning at 14B scale

vs others: Provides reasoning transparency comparable to larger models (GPT-4) while remaining 3-5x cheaper and faster, though with slightly lower accuracy on edge cases