Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation

2026-07-24 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study evaluated two small open-weight Vision-Language Models (VLMs), Qwen2-VL-2B-Instruct and SmolVLM-Instruct, under six realistic photographic degradations at three severity levels. Researchers compared the models' stated confidence in natural language against their internal mean token probability. A significant gap was found: Qwen2-VL's verbalized confidence (mean 0.87-0.90) detected errors at chance (AUROC 0.39-0.75), while its internal token probability achieved high error detection (AUROC 0.92-0.99). SmolVLM's verbalized confidence was largely unobtainable, but its internal probability showed above-chance error detection (AUROC 0.54-0.92). Both models failed under severe underexposure, where accuracy collapsed (Qwen2-VL: 0.99->0.22; SmolVLM: 0.97->0.42) and confidence signals became unreliable. The study concludes small VLMs possess usable self-knowledge not expressed verbally, making internal probability a better deferral signal, though neither is trustworthy in severe low-light.

Key takeaway

For MLOps Engineers deploying small VLMs on consumer hardware, prioritize using the model's internal mean token probability as a deferral signal. Your system's reliability will improve significantly compared to relying on verbalized confidence, which is often inaccurate or unobtainable. However, be aware that under severe low-light conditions, neither confidence signal is trustworthy, requiring alternative error handling or human review.

Key insights

Small VLMs possess internal self-knowledge about errors that their verbalized confidence often fails to express.

Principles

• Internal token probability is a superior VLM error detection signal.
• Verbalized confidence from small VLMs is frequently unreliable.
• Severe underexposure significantly degrades VLM accuracy and confidence.

Method

The study evaluated Qwen2-VL-2B-Instruct and SmolVLM-Instruct across six photographic degradations, comparing natural language stated confidence with mean token probability for error detection.

In practice

• Prioritize internal token probability for VLM deferral mechanisms.
• Do not rely on verbalized confidence from small VLMs.
• Implement specific safeguards for VLMs in low-light conditions.

Topics

• Vision-Language Models
• Model Confidence
• Image Degradation
• Error Detection
• Qwen2-VL
• SmolVLM
• Uncertainty Quantification

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation — Small VLMs possess internal self-knowledge about errors that their verbalized confidence does not express.
• Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation — Model scale improves internal VLM uncertainty signals more than verbalized confidence, while 4-bit quantization degrades confidence significantly.
• Semantic Robustness Certification for Vision-Language Models — This framework certifies Vision-Language Model robustness against semantic variations using text prompts as proxies.
• Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models — VLM reliability is primarily predicted by generation dynamics and internal consistency, not visual attention.
• Test-Time Hinting for Black-Box Vision-Language Models — Test-Time Hinting improves VLM accuracy by preemptively guiding black-box models away from predictable failure modes with a single API call.
• Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation — LLMs' verbalized confidence in translation outputs can be as reliable as internal signals, despite no correlation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.