Squad V2 Benchmark Aligned Evaluation With Unanswerable Question Handling

via “question-answer pair dataset curation and versioning”

Multi-turn conversation benchmark — 80 questions, 8 categories, GPT-4 as judge.

Unique: Explicitly structures questions as multi-turn conversations (not single-turn), with each question containing 2-3 sequential turns that build on prior context. Questions are manually curated by LMSYS researchers rather than automatically generated, ensuring semantic diversity and avoiding trivial or duplicate questions.

vs others: More rigorous than auto-generated benchmarks (HELM uses templates) but smaller in scale; provides explicit multi-turn structure that single-turn benchmarks (MMLU, ARC) cannot evaluate.

via “factuality-benchmark-evaluation-with-unambiguous-answers”

OpenAI's factuality benchmark for hallucination detection.

Unique: Focuses specifically on unambiguous factual questions where ground truth is objectively determinable, eliminating subjective evaluation variance that plagues other factuality benchmarks; uses OpenAI's curation process to ensure questions have single correct answers with no reasonable interpretation ambiguity

vs others: More precise than general QA benchmarks (SQuAD, TriviaQA) because it explicitly filters for unambiguous answers, making hallucination detection clearer and more actionable than benchmarks that tolerate multiple valid responses

via “adversarial unanswerable question generation and validation”

150K reading comprehension questions including unanswerable ones.

Unique: Pioneered adversarial unanswerable questions in QA benchmarks by having crowdworkers explicitly write questions that CANNOT be answered from a passage. This is fundamentally different from randomly sampling unanswerable questions; adversarial construction ensures questions are plausible but genuinely unanswerable.

vs others: More challenging than datasets with random negative examples (e.g., MS MARCO) because adversarial questions require models to understand semantic relevance, not just keyword matching, to distinguish answerable from unanswerable.

via “answerability classification with unanswerable question handling”

307K real Google Search queries answered from Wikipedia.

Unique: Explicitly includes unanswerable questions with labels rather than filtering them out, forcing systems to learn rejection as a valid output rather than always attempting answer extraction

vs others: More realistic than QA benchmarks that only include answerable questions, and directly addresses the hallucination problem that production systems face

via “grade-school science question benchmark evaluation”

7.8K science questions testing genuine reasoning, not just recall.

Unique: Explicitly designed to filter out questions answerable by retrieval or word co-occurrence — the Challenge subset (2,590 questions) was curated by removing questions that simple baseline methods could solve, ensuring the remaining questions require genuine multi-step reasoning and knowledge application rather than surface-level pattern matching

vs others: More rigorous than generic QA benchmarks because it explicitly excludes questions solvable by shallow methods, making it a stricter test of reasoning; smaller and more focused than MMLU but with deeper curation for reasoning-specific evaluation

via “standardized answer extraction and correctness comparison”

8.5K grade school math problems — multi-step reasoning, verifiable solutions, reasoning benchmark.

Unique: Uses a simple, language-agnostic delimiter format (####) for answer marking that works across any model output format, combined with numeric comparison logic that handles floating-point precision and integer equivalence, enabling consistent evaluation without model-specific parsing

vs others: More robust than regex-based answer extraction (explicit delimiter is unambiguous) and more scalable than manual evaluation, but less sophisticated than semantic similarity metrics that could credit partially correct reasoning

via “benchmark-validated reasoning performance on standardized datasets”

Alibaba's 32B reasoning model with chain-of-thought.

Unique: Provides documented benchmark results on standardized reasoning datasets (AIME 79.5%, MATH-500 96.4%) enabling quantitative performance validation, with explicit comparison claims against larger models

vs others: Demonstrates competitive reasoning performance on standardized benchmarks comparable to much larger models, providing quantitative evidence of reasoning capability for evaluation and comparison purposes

via “benchmark evaluation suite for ocr-vqa model performance”

45K questions requiring reading text in images.

Unique: Evaluation framework explicitly measures the intersection of OCR and reasoning capabilities by requiring models to both detect/recognize text AND answer questions about it, rather than evaluating these as separate tasks; provides structured comparison across models with different OCR backends (learned vs. traditional)

vs others: More rigorous than ad-hoc evaluation because it uses a fixed, large-scale benchmark with standardized splits, but less flexible than custom evaluation scripts that can measure task-specific metrics like OCR token-level F1 or reasoning accuracy in isolation

via “comprehensive model evaluation and benchmarking”

Tiny vision-language model for edge devices.

Unique: Comprehensive evaluation suite covering VQA (accuracy), document understanding (DocVQA metrics), chart analysis (ChartQA), and real-world QA with reference implementations for each benchmark; integrates scoring utilities that compute BLEU, CIDEr, and accuracy metrics without external dependencies.

vs others: Integrated evaluation framework reduces setup friction compared to manual benchmark implementation; covers multiple task types (VQA, document, chart) in single codebase, enabling holistic model assessment.

via “squad v2 benchmark-aligned evaluation with unanswerable question handling”

question-answering model by undefined. 6,23,377 downloads.

Unique: Explicitly trained on SQuAD v2's unanswerable questions subset, learning to recognize when no valid answer exists rather than always extracting a span — unlike SQuAD v1-only models that lack this capability and will hallucinate answers for out-of-scope questions

vs others: More reliable than v1-trained models in production because it can admit when it doesn't know, reducing false positive answers and improving user trust in systems that route unanswerable questions to humans

via “squad 2.0 unanswerable question detection”

question-answering model by undefined. 2,87,434 downloads.

Unique: Trained on SQuAD 2.0's adversarial unanswerable questions, learning to distinguish answerable from unanswerable via the same span prediction mechanism rather than a separate binary classifier. This is more parameter-efficient but less explicit than dedicated answerability heads.

vs others: More robust to unanswerable questions than SQuAD 1.1-only models because it was explicitly trained on adversarial non-answers, reducing hallucination on out-of-scope queries.

via “adversarial no-answer detection via binary classification head”

question-answering model by undefined. 8,99,590 downloads.

Unique: Explicitly trained on SQuAD 2.0's adversarial no-answer examples (human-written questions that appear answerable but have no correct answer in the passage), giving it a specialized capability to reject unanswerable questions rather than extracting incorrect spans. This is a distinct training objective from standard SQuAD 1.1 models.

vs others: More robust to adversarial no-answer cases than BERT-base QA models trained only on SQuAD 1.1, but requires careful threshold tuning and may not generalize to no-answer patterns outside SQuAD 2.0's distribution.

via “multimodal question-answering evaluation”

Visual Question Answering with real images and human questions

Unique: VQAv2 combines a large-scale dataset with a diverse range of question types, enabling comprehensive evaluation of vision-language models, unlike simpler datasets that may focus on a narrower scope.

vs others: More comprehensive than other visual question-answering benchmarks due to its extensive question variety and large image corpus.

via “squad v2 benchmark-aligned answer span prediction”

question-answering model by undefined. 1,93,069 downloads.

Unique: Trained on SQuAD v2's 50k unanswerable questions (vs. SQuAD v1 which had only answerable questions), exposing the model to negative examples where the answer is not in the passage, improving robustness to out-of-distribution queries

vs others: Achieves ~88-90 F1 on SQuAD v2 dev set (competitive with BERT-large baseline); better calibrated confidence scores than SQuAD v1-only models due to unanswerable question exposure

via “squad-optimized span classification with confidence scoring”

question-answering model by undefined. 1,16,670 downloads.

Unique: Trained on SQuAD v1.1 with contrastive negative sampling to learn span boundaries precisely, producing calibrated confidence scores that correlate with answer correctness — not just raw logits, but post-processed probabilities validated on held-out SQuAD test set

vs others: Achieves 88.5% F1 on SQuAD v1.1 (vs 91% for full BERT-base) while being 40% faster, and provides confidence scores out-of-the-box without requiring separate uncertainty quantification layers

via “squad 2.0 benchmark evaluation and metric computation”

question-answering model by undefined. 1,45,572 downloads.

Unique: Trained on SQuAD 2.0 with published benchmark results (EM: 76.8%, F1: 84.6%) enabling direct comparison against other models on the same dataset, with explicit handling of unanswerable questions in metric computation

vs others: Smaller model size achieves competitive SQuAD 2.0 performance compared to larger models (BERT-base, ELECTRA), making it suitable for resource-constrained deployments without sacrificing benchmark accuracy

via “squad-v2-optimized span boundary detection”

question-answering model by undefined. 3,19,759 downloads.

Unique: Explicitly trained on SQuAD v2's 30% unanswerable questions with negative sampling, enabling the model to learn when to output null predictions rather than forcing spurious span selections — a critical capability absent in v1-only models

vs others: More robust than SQuAD v1-trained models on real-world QA because it has learned to recognize and correctly handle unanswerable questions, reducing false-positive answer predictions in production systems

via “squad 2.0-compatible unanswerable question detection”

question-answering model by undefined. 1,90,899 downloads.

Unique: Trained on SQuAD 2.0's adversarial unanswerable questions (33% of dataset), learning to predict null spans rather than forcing answers from irrelevant text; uses disentangled attention to better distinguish between answerable and unanswerable contexts

vs others: Achieves 88%+ F1 on SQuAD 2.0 unanswerable detection vs 75-80% for models fine-tuned only on SQuAD 1.1, reducing false-positive answer hallucinations in production systems

via “adversarial unanswerable question detection”

question-answering model by undefined. 1,24,380 downloads.

Unique: SQuAD v2 training includes 30% adversarial unanswerable examples written by humans to trick extractive models, enabling robust null prediction vs SQuAD v1 models that assume all questions are answerable

vs others: Provides built-in unanswerable detection without separate classifier, reducing latency vs ensemble approaches; more robust than simple confidence thresholding due to adversarial training

via “unanswerable question detection with confidence scoring”

question-answering model by undefined. 32,657 downloads.

Unique: SQuAD v2 training includes adversarially-written unanswerable questions (plausible but incorrect passages) rather than random negatives, forcing the model to learn semantic mismatch detection. MobileBERT preserves this capability through its [CLS] token 'no answer' head, enabling robust abstention without post-hoc filtering.

vs others: More reliable unanswerable detection than SQuAD v1-only models due to adversarial training data; comparable to full BERT-base but with 5.5x faster inference, making it practical for real-time filtering in retrieval pipelines.