MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

MissingBench-Verified, a new benchmark, reveals a fundamental limitation in ten leading Vision-Language Models (VLMs) regarding their ability to detect missing essential object parts. The study found consistent and significant failure rates, with all tested models achieving less than 75% accuracy in identifying absent components. Claude Sonnet 4.6 recorded the lowest accuracy at 44.1%, while Qwen 3.5 Flash frequently asserted missing objects were "Clearly Visible" in 51.7% of trials, exceeding its correct "Not Visible" detections (47.5%). Even when provided with external tool evidence, such as simulated perfect object detection results, or access to image processing tools like cropping and contrast adjustment, most models showed negligible improvement. Gemini 3.1 Pro Preview was a notable exception, reaching 82.2% accuracy with post-answer tool injection. Increasing reasoning effort in GPT-5.2 also failed to mitigate these hallucinations, suggesting they are deeply embedded in the models' parametric knowledge rather than superficial reasoning errors. Object detection models like OWLv2 and YOLOE also struggled, exhibiting different failure modes related to detection capability rather than hallucination.

Key takeaway

For AI Engineers deploying Vision-Language Models in critical inspection or monitoring systems, you must recognize their inherent inability to reliably detect missing essential object components. Your current prompting strategies, tool integration, or extended reasoning efforts will likely not resolve these deep-seated hallucinations. Prioritize VLM architectures or training interventions that explicitly enable models to override strong internal expectations with contradictory visual evidence, rather than relying on post-hoc corrections.

Key insights

VLMs fundamentally struggle to detect missing essential object parts due to deeply embedded prior knowledge, resisting current mitigation strategies.

Principles

• VLMs prioritize internal priors over contradictory visual evidence.
• Hallucinations are deeply embedded, not superficial reasoning errors.
• Current post-hoc mitigation strategies are largely ineffective.

Method

MissingBench-Verified constructs a dataset of 118 human-verified images with essential parts removed, evaluating VLMs via multiple-choice questions under direct, tool-injected, and image processing tool settings.

In practice

• Re-evaluate VLMs for critical inspection tasks.
• Prioritize architectural changes for VLM robustness.
• Develop training data for objects with missing parts.

Topics

• Vision-Language Models
• Object Hallucination
• MissingBench-Verified
• Model Robustness
• Inspection Systems
• Parametric Knowledge

Best for: Research Scientist, Computer Vision Engineer, AI Product Manager, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models — Hallucinated semantics in VLMs can surprisingly improve vision-language task accuracy by broadening semantic coverage during post-hallucination reasoning.
• CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework — CaVe-VLM-CoT enforces evidence-grounded reasoning in VLMs through a closed-loop, reflection-based RAG agent to combat hallucinations.
• LADBench: A Benchmark for Logical Fault Detection in Images — VLMs struggle with logical fault detection in images, even with explicit hints, highlighting a gap in common sense reasoning.
• Hallucination Detection and Correction in Medical VLMs via Counter-Evidence Verification — CoEV verifies VLM outputs against visual evidence to detect and correct medical hallucinations.
• Diagnosing Visual Ignorance in Vision-Language Models — VLMs' visual ignorance results from internal routing failures and benchmarks that reward language-prior reliance.
• GAVEL: Grounded Caption Error Verification and Localization — GAVEL unifies VLM error verification, explanation, and visual localization into a single, learnable task.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.