Logical Deduction And Inference Evaluation

23 hardest BIG-Bench tasks where models initially failed.

Unique: Isolates formal logical reasoning as a distinct capability by presenting logic problems in natural language with few-shot examples, testing whether models can apply logical rules consistently without explicit training. This approach measures logical inference generalization.

vs others: More focused on formal logical reasoning than general reasoning benchmarks; more accessible than formal logic verification because it uses natural language rather than symbolic logic notation.