Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts

2026-07-10 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study on "Silent Failures in Quantized LLM Reasoning" reveals that post-training quantization can subtly alter how large language models reason, even when overall task accuracy is maintained. Researchers classified 30,000 chain-of-thought outputs from five instruction-tuned LLMs (3B-14B parameters) across FP32, FP16, and NF4 precisions using a six-category failure taxonomy. While accuracy dropped by a maximum of 3.1 percentage points, "Hollow Convergence"—correct answers derived from incomplete reasoning—showed significant size-dependent shifts under NF4, sharply decreasing for models below 12B parameters but remaining stable for larger models. This effect varied by benchmark, with GSM8K being immune while LogiQA and ARC-Challenge exhibited the largest shifts. Furthermore, for LLaMA 3.2-3B under NF4, "Shortcut Collapse" increased from 44% to 78% of wrong-answer failures, and "Confidence Snowballing" nearly vanished from 15.8%. Critically, Hollow Convergence is difficult to detect from surface-level text, achieving a best F1 score of 0.53.

Key takeaway

For MLOps Engineers deploying quantized LLMs, relying solely on accuracy metrics is insufficient. You should implement advanced evaluation pipelines that specifically detect "Hollow Convergence" and analyze shifts in failure modes, especially when using NF4 quantization on models smaller than 12B parameters. Your current evaluation methods likely miss critical reasoning degradations, necessitating deeper qualitative analysis to ensure robust model behavior in production.

Key insights

Post-training quantization can silently degrade LLM reasoning quality, even with preserved task accuracy, through "Hollow Convergence" and failure mode shifts.

Principles

• Quantization effects on reasoning are size-dependent, impacting smaller LLMs more.
• Specific benchmarks exhibit varying susceptibility to quantization-induced reasoning shifts.
• Surface-level text features are insufficient to detect "Hollow Convergence."

Method

A six-category failure taxonomy, validated by human annotators (Cohen's κ = 0.906), was used to classify 30,000 chain-of-thought outputs from LLMs across different quantization precisions and reasoning benchmarks.

In practice

• Evaluate quantized LLMs beyond accuracy, specifically for "Hollow Convergence."
• Consider model size and benchmark type when applying NF4 quantization.

Topics

• LLM Quantization
• Reasoning Failures
• Hollow Convergence
• NF4 Precision
• Chain-of-Thought
• Model Evaluation

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty — Structural uncertainty quantifies LLM logical reasoning consistency by measuring the stability of self-preference-induced rankings over sampled solutions.
• Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment — KL divergence tracks disagreement volume, not direction, making it unreliable for ranking near-baseline quantized LLMs.
• QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation — QMFOL provides a scalable framework for generating controllable, complex monadic first-order logic reasoning benchmarks for LLMs.
• How Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures — LLM reasoning failures manifest as "committed" (early lock-in) or "persistent" (accumulating uncertainty), detectable via token-level signals.
• The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs — Quantization in LLMs creates behavioral divergence not captured by accuracy, necessitating new evaluation metrics.
• The Spectral Illusion: Why Fixing “Representation Collapse” Won’t Save LLMs from Themselves — Architectural fixes for LLM representation collapse are sophisticated band-aids, failing to address the fundamental limits of statelessness and true reasoning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.