Mechanisms of Prompt-Induced Hallucination in Vision-Language Models

2026-04-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Vision-Language Models, Model Interpretability · Depth: Expert, extended

Summary

A study by Rudman et al. investigates prompt-induced hallucinations (PIH) in large vision-language models (VLMs), where models prioritize textual prompts over conflicting visual evidence. Using a controlled object-counting task with misaligned prompts (e.g., asking for four waterlilies when only three are present), the researchers found that VLMs often correct overestimations at low object counts but increasingly conform to the prompt as the number of objects increases, even with large discrepancies. Through mechanistic analysis of three VLMs (LLaVA-OneVision-7B, Qwen2-VL-7B, and Janus-Pro-7B), the study identified a small set of attention heads, termed PIH-heads, whose ablation significantly reduces hallucinations by at least 40% without additional training. These PIH-heads, primarily located in early language model layers, mediate prompt copying and, when ablated, increase reliance on visual evidence, generalizing to tasks beyond counting, such as color identification, with up to a 94.25% reduction in prompt-color copying.

Key takeaway

For AI Engineers and Research Scientists developing or deploying VLMs, understanding and mitigating prompt-induced hallucinations is critical for reliability. You should consider implementing targeted attention head ablations, particularly in early language model layers, to reduce text-over-vision bias. This approach can significantly improve visual grounding and factual accuracy without requiring extensive retraining, enhancing model robustness in real-world applications with potentially noisy inputs.

Key insights

VLMs hallucinate by prioritizing text over vision, a behavior traceable to specific, ablatable attention heads.

Principles

• PIH increases with object count.
• PIH-heads are concentrated in early LM layers.
• Ablation shifts reliance to visual evidence.

Method

Identify PIH-mediating attention heads via mean ablation, then ablate these heads to reduce prompt-induced hallucinations and enhance visual grounding.

In practice

• Ablate PIH-heads to reduce VLM hallucinations.
• Focus on early LM layers for intervention.
• Test ablation effects across diverse tasks.

Topics

• Vision-Language Models
• Prompt-Induced Hallucinations
• Attention Head Ablation
• Mechanistic Interpretability
• Object Counting Task

Code references

• michalg04/prompt-induced_hallucinations

Best for: AI Engineer, Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs — LVLM hallucinations are largely driven by over-reliance on textual instruction priors, not just vision backbone limits.
• VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck — VIB-Probe uses Variational Information Bottleneck theory to detect and mitigate VLM hallucinations by analyzing internal attention head outputs.
• Causal Evidence for Attention Head Imbalance in Modality Conflict Hallucination — MLLM modality conflict arises from an imbalanced attention head routing structure, addressable by targeted intervention.
• Comparing Human Gaze and Vision-Language Model Attention in Safety-Relevant Environments — VLMs can effectively mimic human visual attention in safety-critical scenes, bypassing eye-tracking data needs.
• Reducing Hallucination in Vision-Language Models via Stage-wise Preference Optimization under Distribution Shift — Stage-wise preference optimization with targeted data construction effectively reduces VLM hallucination by focusing on difficult grounding boundaries.
• Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps — Attention-derived metrics enable efficient, inference-time hallucination detection in SpeechLLMs without gold-standard outputs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.