Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation

2026-07-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

A study evaluated two small open-weight Vision-Language Models, Qwen2-VL-2B-Instruct and SmolVLM-Instruct, under six realistic photographic degradations at three severity levels. Researchers compared two confidence signals: verbalized confidence, stated in natural language, and internal token probability, derived from the model's generated answer. Across 3,800 predictions, Qwen2-VL's verbalized confidence remained nearly constant (mean 0.87–0.90) and detected errors at chance level (AUROC 0.39–0.75). In contrast, its internal token probability achieved strong error separation (AUROC 0.92–0.99). SmolVLM struggled to produce parseable verbalized confidence, but its internal probability showed above-chance error detection (AUROC 0.54–0.92). Both models failed under severe underexposure, with accuracy collapsing (e.g., Qwen2-VL from 0.99 to 0.22) while confidence signals remained stable, rendering error detection at chance. The findings suggest small VLMs possess internal self-knowledge not reflected in their verbal output, making internal probability a superior deferral signal, though neither is reliable in severe low-light conditions.

Key takeaway

For MLOps Engineers deploying small Vision-Language Models in environments with degraded image inputs, you should prioritize internal token probability over verbalized confidence for uncertainty estimation and error deferral. Your models' stated confidence is likely overconfident and fails to track actual reliability, whereas internal token probabilities offer significantly better error detection (AUROC up to 0.99). However, be aware that under severe underexposure, both confidence signals become unreliable, requiring alternative error handling or human intervention.

Key insights

Small VLMs possess internal self-knowledge about errors that their verbalized confidence does not express.

Principles

• Internal token probability is a more reliable error detection signal than verbalized confidence.
• Verbalized confidence in small VLMs often remains constant and overconfident.
• Neither confidence signal is reliable under severe low-light conditions.

Method

Evaluate small open-weight VLMs (Qwen2-VL-2B-Instruct, SmolVLM-Instruct) on a 100-item Food101 subset with six realistic photographic degradations at three severity levels, comparing verbalized confidence and internal token probability via AUROC.

In practice

• Prioritize internal token probability for deferral mechanisms in constrained VLM deployments.
• Avoid relying on verbalized confidence from small VLMs for uncertainty estimation.
• Implement specific checks for severe underexposure, as confidence signals fail here.

Topics

• Small Vision-Language Models
• Uncertainty Estimation
• Image Degradation
• Verbalized Confidence
• Internal Token Probability
• Error Detection

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation — Small VLMs possess internal self-knowledge about errors that their verbalized confidence often fails to express.
• Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation — Model scale improves internal VLM uncertainty signals more than verbalized confidence, while 4-bit quantization degrades confidence significantly.
• Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language — LLMs consistently misinterpret probabilistic information, particularly uncertainty, when generating natural language explanations, indicating a verbalization bottleneck.
• Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models — MLLMs significantly improve scene description accuracy on complex social behaviors, nearly matching human performance and eliminating most error types.
• CALIBER: Calibrating Confidence Before and After Reasoning in Language Models — Confidence in reasoning language models is state-dependent, requiring distinct supervision targets for pre- and post-reasoning estimates.
• Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias — VLMs for medical image quality assessment exhibit significant vulnerabilities to image corruption and contextual metadata bias.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.