Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

2026-07-27 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Language model benchmarks often obscure critical failure modes by collapsing two distinct measurement questions—whether a response reached an evaluable state and whether its answer was correct—into a single accuracy score. A new two-layer evaluation framework is introduced to separate scorer-independent execution evidence, such as termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits revealed sharply different execution mixtures. For instance, 49 of 450 Qwen MATH outputs terminated without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Furthermore, a coverage-audited study showed that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates, demonstrating that accuracy conflates execution case mix with verification policy.

Key takeaway

For machine learning engineers evaluating language models or designing new benchmarks, relying solely on aggregate accuracy scores is insufficient. You should implement a two-layer evaluation framework that explicitly separates execution evidence, such as termination and parseability, from correctness judgments. This approach will reveal critical failure modes hidden by traditional metrics, enabling more robust model comparisons and targeted improvements in model reliability and output quality.

Key insights

Language model accuracy conflates execution states with correctness, masking critical failure modes in benchmarks.

Principles

• Separate execution evidence from correctness.
• Token limits impact LM output execution states.
• Verification policies alter accuracy estimates.

Method

Implement a two-layer evaluation framework separating scorer-independent execution evidence (termination, parseability, length) from scorer-dependent correctness.

In practice

• Report pre-intervention execution states.
• Document verification coverage.
• Include scorer provenance details.

Topics

• Language Model Evaluation
• Benchmark Design
• Execution States
• Qwen Models
• DeepSeek Models
• Token Limits

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• How Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures — Language model reasoning failures leave distinct token-level uncertainty signatures, enabling early detection or full-trace analysis.
• Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models — Task competence and instruction following are distinct abilities, with standard accuracy hiding instruction-following failures.
• Systematic Performance Degradation in Indic Vision-Language Models: Evidence from Hindi and Telugu — Multilingual VLM performance systematically degrades in Indic languages due to current benchmark limitations and lack of cultural grounding.
• Everyone Is Chasing AI Benchmarks, Almost Nobody Is Measuring Truth — Traditional AI benchmarks measure ability, not factual truth, leading to misplaced trust in model outputs.
• ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence — ConflictScore measures how language models acknowledge conflicting evidence, improving truthfulness and detecting overconfidence.
• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.