Algorithmic Reasoning Task Evaluation

via “solution step extraction and intermediate reasoning evaluation”

12.5K competition math problems — AMC/AIME/Olympiad level, 7 subjects, standard math benchmark.

Unique: Preserves solution steps as first-class data throughout the evaluation pipeline, enabling evaluation of intermediate reasoning quality rather than just final answers. This supports emerging research on chain-of-thought prompting and interpretable AI reasoning.

vs others: More comprehensive than final-answer-only evaluation because it assesses reasoning quality and interpretability, but requires more manual annotation and is harder to automate than simple answer verification.

via “logical deduction task evaluation”

Zero-shot LLM evaluation for reasoning tasks.

Unique: Provides unified evaluation framework for both symbolic logic and natural language reasoning puzzles in zero-shot setting, with answer verification that can handle both formal symbolic validation and semantic similarity-based matching for natural language conclusions

vs others: More specialized than general reasoning benchmarks; focuses specifically on logical deduction without few-shot examples, enabling cleaner measurement of foundational logical capability vs. pattern-matching from examples

via “abstract-pattern-recognition-evaluation”

Abstract reasoning benchmark with $1M prize for AGI.

Unique: Explicitly designed to measure learning efficiency and abstract reasoning on novel tasks, resisting scaling-only solutions. Foundation claims 'scaling alone will not reach AGI' and positions ARC-AGI as identifying capability gaps that require new algorithmic ideas, not just parameter scaling.

vs others: Differs from knowledge benchmarks (MMLU, TriviaQA) by requiring genuine learning and generalization rather than retrieval; differs from domain-specific reasoning benchmarks (math, code) by using abstract visual puzzles without domain conventions or pre-training advantages.

23 hardest BIG-Bench tasks where models initially failed.

Unique: Isolates algorithmic reasoning as a distinct capability by presenting algorithm problems in natural language with few-shot examples, testing whether models can learn algorithmic patterns without explicit training. This approach measures algorithmic reasoning generalization rather than memorization.

vs others: More focused on algorithmic reasoning than general reasoning benchmarks; more accessible than formal algorithm verification tasks because it uses natural language rather than pseudocode or formal logic.

via “reasoning and chain-of-thought decomposition for complex tasks”

Google's open-weight model family from 1B to 27B parameters.

Unique: 27B variant achieves reasoning performance competitive with much larger models (70B+) through optimized training on reasoning-heavy datasets and learned chain-of-thought patterns, without requiring external reasoning engines or symbolic solvers

vs others: Outperforms Llama 2 70B on math and coding reasoning benchmarks while being 2.6x smaller, and matches Mistral 7B on reasoning tasks while offering superior code generation quality

via “reasoning and multi-step problem decomposition”

TII's 180B model trained on curated RefinedWeb data.

Unique: Achieves strong reasoning performance through scale (180B parameters) and data quality (3.5T meticulously-cleaned RefinedWeb tokens) rather than specialized reasoning fine-tuning, enabling emergent reasoning capabilities across diverse domains without task-specific training.

vs others: Larger parameter count than reasoning-specialized models like Llama 2 70B enables better few-shot reasoning, but lacks explicit chain-of-thought fine-tuning that models like GPT-4 or Claude employ, potentially requiring more sophisticated prompting to achieve comparable reasoning quality.

via “reasoning difficulty stratification (easy vs. challenge)”

7.8K science questions testing genuine reasoning, not just recall.

Unique: Challenge subset was explicitly curated by removing questions answerable by retrieval-based and word co-occurrence baseline methods, rather than using heuristic difficulty metrics. This ensures that Challenge questions genuinely require reasoning beyond surface-level pattern matching, making it a more rigorous test of reasoning capability than difficulty-sorted datasets.

vs others: More principled than arbitrary difficulty splits because curation is based on empirical baseline performance; more focused on reasoning than datasets that use question length or vocabulary complexity as difficulty proxies

via “multi-step mathematical reasoning benchmark evaluation”

8.5K grade school math problems — multi-step reasoning, verifiable solutions, reasoning benchmark.

Unique: Uses linguistically diverse, human-authored grade school problems (not synthetic) that require genuine multi-step reasoning with basic arithmetic, combined with a standardized answer extraction format (#### delimiter) that enables reproducible evaluation across heterogeneous model outputs

vs others: More challenging than simple arithmetic benchmarks (requires 2-8 reasoning steps) yet more accessible than advanced math benchmarks, making it ideal for measuring practical reasoning improvements in production models

via “algorithmic reasoning and complexity assessment”

10K coding problems across 3 difficulty levels with test suites.

Unique: Explicitly sources problems from competitive programming platforms (AtCoder, Codeforces, Kattis) where algorithmic rigor and time/memory limits enforce genuine complexity requirements, rather than using toy problems that can be solved with naive approaches

vs others: Tests genuine algorithmic reasoning rather than API knowledge; problems cannot be solved by simple pattern matching or memorization, requiring models to understand data structures, complexity analysis, and algorithm selection

via “arc-agi benchmark reasoning and abstract problem-solving”

OpenAI's most powerful reasoning model for complex problems.

Unique: Achieves 87.5% on ARC-AGI through extended reasoning about visual-logical patterns and rule inference, exploring multiple hypotheses about transformation rules before committing to predictions — this reasoning-first approach outperforms pattern-matching baselines

vs others: Significantly outperforms GPT-4 and Claude on ARC-AGI (87.5% vs ~50-60%) by allocating extended reasoning to hypothesis formation and rule inference rather than direct pattern matching, demonstrating genuine abstract reasoning capability

via “abstract reasoning and pattern recognition (arc-agi)”

Google's most capable model with 1M context and native thinking.

Unique: Extended thinking enables exploration of multiple pattern hypotheses before settling on final answer; achieves 77.1% on ARC-AGI-2 through genuine reasoning rather than memorized patterns

vs others: Significantly outperforms GPT-4 (unknown ARC score) and Claude 3.5 Sonnet (58.3% ARC-AGI-2) on abstract reasoning; better at generalizing from limited examples

via “evaluation metric formulation”

Abstraction and reasoning corpus for general intelligence

Unique: The evaluation metrics are specifically tailored to assess abstract reasoning capabilities, unlike generic metrics that may not reflect reasoning depth.

vs others: Offers more nuanced evaluation than traditional benchmarks like accuracy, which may not fully capture reasoning abilities.

via “commonsense reasoning evaluation”

Commonsense NLI with adversarial context mining

Unique: Utilizes adversarially filtered questions to create plausible distractors, ensuring a more robust evaluation of reasoning capabilities compared to traditional benchmarks.

vs others: More challenging than standard commonsense benchmarks due to its focus on plausible distractors, making it a better test for true understanding.

via “reasoning capability evaluation”

Subset of BIG-Bench where most models fail

Unique: The curation of tasks specifically targeting reasoning limits rather than general performance allows for a more focused evaluation of model capabilities.

vs others: More targeted than generic benchmarks, as it specifically identifies and tests reasoning weaknesses in models.

via “mathematical reasoning and logic problem evaluation with specialized scoring”

ReLE评测：中文AI大模型能力评测（持续更新）：目前已囊括374个大模型，覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型， 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜，也提供规模超200万的大

Unique: Evaluates mathematical reasoning with 1-5 quality scale for reasoning steps rather than binary correctness, enabling partial credit for correct methodology with computational errors. Combines final answer accuracy with reasoning quality assessment to capture mathematical thinking capability. Includes multi-step reasoning problems and logical inference tasks beyond simple arithmetic.

vs others: More nuanced mathematical assessment than MMLU (binary correctness) and captures reasoning quality vs answer-only evaluation

via “reasoning and step-by-step problem decomposition”

Gemma 4 26B A4B IT is an instruction-tuned Mixture-of-Experts (MoE) model from Google DeepMind. Despite 25.2B total parameters, only 3.8B activate per token during inference — delivering near-31B quality at...

Unique: MoE expert specialization enables dedicated reasoning experts that activate for complex reasoning tasks, while general-purpose experts handle simpler steps, optimizing compute allocation across reasoning complexity

vs others: Provides faster reasoning than Llama 3.1 8B (15-20% speedup) while maintaining comparable accuracy on grade-school math and logic puzzles, though underperforms specialized reasoning models like o1-mini on competition-level problems

via “reasoning and step-by-step problem decomposition”

Meta's latest class of model (Llama 3.1) launched with a variety of sizes & flavors. This 70B instruct-tuned version is optimized for high quality dialogue usecases. It has demonstrated strong...

Unique: Instruction-tuned on datasets containing explicit reasoning traces (e.g., math solutions with working, logic puzzles with step-by-step explanations), enabling the model to learn to generate intermediate reasoning as a learned behavior rather than relying on prompt engineering alone.

vs others: More reliable than base models at producing coherent reasoning chains; comparable to GPT-4 on standard benchmarks but with lower latency and cost, though may underperform on novel reasoning patterns not well-represented in training data.

via “extended reasoning with implicit chain-of-thought”

Grok 4 is xAI's latest reasoning model with a 256k context window. It supports parallel tool calling, structured outputs, and both image and text inputs. Note that reasoning is not...

Unique: Implicit reasoning allocation based on problem complexity, with reasoning traces integrated into output without explicit token budget management, contrasting with OpenAI's explicit reasoning token approach

vs others: More transparent reasoning than GPT-4o (which hides reasoning) but less controllable than o1 (which offers explicit reasoning token budgets); better for exploratory reasoning where depth is problem-dependent

via “reasoning and chain-of-thought decomposition”

Gemma 4 26B A4B IT is an instruction-tuned Mixture-of-Experts (MoE) model from Google DeepMind. Despite 25.2B total parameters, only 3.8B activate per token during inference — delivering near-31B quality at...

Unique: Reasoning capability emerges from instruction-tuning on datasets containing reasoning examples, not explicit reasoning modules or symbolic reasoning engines. The model learns to generate plausible reasoning chains through imitation, making it flexible but not formally verifiable.

vs others: Provides comparable chain-of-thought quality to GPT-4 on most reasoning tasks while using 3x fewer active parameters, though may require more explicit prompting to trigger reasoning compared to larger models.

via “reasoning and step-by-step problem solving”

Olmo 3.1 32B Instruct is a large-scale, 32-billion-parameter instruction-tuned language model engineered for high-performance conversational AI, multi-turn dialogue, and practical instruction following. As part of the Olmo 3.1 family, this...

Unique: Instruction-tuning on chain-of-thought datasets enables the model to generate coherent reasoning steps when prompted, without requiring explicit reasoning modules or external symbolic solvers — this implicit reasoning approach is more flexible than hard-coded reasoning systems but less precise than specialized solvers

vs others: More transparent reasoning than direct answer generation, but lower accuracy on specialized domains than models fine-tuned exclusively on reasoning tasks; better for educational use cases than production problem-solving