HalluTrace: Causal Attribution and Source-Targeted Decoding for Hallucination in Large Vision-Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

HALLUTRACE is a causal attribution framework designed to understand object hallucination mechanisms in large vision-language models (LVLMs). It decomposes hallucination into three distinct sources: visual grounding failure (VGF), language prior dominance (LPD), and cross-modal conflict (CMC). These sources are operationalized via causal component ablations on fvis, fproj, and fLM, measuring CHAIR score changes. Experiments on five LVLMs show object-category-specific and model-consistent patterns; person/vehicle hallucinations are predominantly LPD (≥52%), while food/furniture are mainly VGF (≥44%). Guided by these attributions, Hallucination-Aware Decoding (HAD) applies source-targeted interventions, such as visual signal amplification for VGF and language prior suppression for LPD. HAD reduces CHAIRI by 3.7–5.6 points and improves POPE F1 by 1.9–3.1 points over LLaVA-1.5, outperforming VCD and ICD on CHAIR, POPE, and MME benchmarks without additional training. The CHAIR improvement from HAD is linearly predictable from the VGF attribution share (r = 0.86, p < 10−6).

Key takeaway

For Machine Learning Engineers deploying or fine-tuning large vision-language models, understanding hallucination sources is critical. You should consider using causal attribution frameworks like HALLUTRACE to diagnose whether visual grounding failures, language prior dominance, or cross-modal conflicts are prevalent for specific object categories. This diagnosis enables you to apply targeted decoding strategies, such as amplifying visual signals or suppressing language priors, to reduce hallucination rates by 3.7–5.6 CHAIRI points without additional model training.

Key insights

HALLUTRACE causally attributes LVLM hallucinations to visual, language, or cross-modal conflicts, enabling targeted decoding strategies.

Principles

• Hallucination sources are object-category-specific.
• Causal attribution can guide targeted decoding.
• Attribution share predicts decoding improvement.

Method

HALLUTRACE operationalizes VGF, LPD, and CMC sources via causal component ablations on fvis, fproj, and fLM, measuring CHAIR score changes. HAD applies visual signal amplification (VGF), language prior suppression (LPD), and contrastive re-weighting (CMC).

In practice

• Amplify visual signals for VGF-prone objects.
• Suppress language priors for LPD-dominant categories.
• Apply contrastive re-weighting for CMC cases.

Topics

• LVLM Hallucination
• Causal Attribution
• Decoding Strategies
• Visual Grounding
• Language Priors
• Cross-Modal Conflict

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• P\textsuperscript{2}-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization — P²-DPO reduces LVLM hallucination by self-generating on-policy visual preference pairs and using a calibration loss.
• Diagnosing Visual Ignorance in Vision-Language Models — Vision-Language Models frequently prioritize language priors over visual evidence due to internal routing failures and flawed benchmark evaluations.
• TPA: Next Token Probability Attribution for Detecting Hallucinations in RAG — SPAD attributes token probabilities to seven sources and aggregates by POS tags to detect RAG hallucinations.
• MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models — Hallucinations in VideoLMMs can be mitigated by selectively patching unreliable visual tokens before language generation.
• Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference — Retrieval-augmented reliability-aware inference significantly reduces visual hallucinations in MLLMs without retraining.
• How Many Counterfactuals Does It Take? Probing VLM Hallucinations Through Circuits and Causal Effects — The study quantifies VLM hallucination robustness using a causal influence metric and circuit discovery.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.