ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs

2026-06-30 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Natural Language Processing · Depth: Expert, quick

Summary

ADAPT (Attention Dynamics Alignment with Preference Tuning) is a new attention-based framework designed to mitigate hallucination in Multimodal Large Language Models (MLLMs). Researchers identified that hallucination stems from a progressive degradation of text-to-image cross-attention during generation, leading to unfocused or biased attention. ADAPT directly intervenes on these cross-attention dynamics through three key contributions: a cross-attention visual anchor, refined from early decoding for stable spatial grounding; an attention-supervised inference mechanism that detects and corrects attention drift online; and a Visual Attention Guidance DPO for aligning preferences towards visually grounded responses. Experiments demonstrate that ADAPT significantly reduces hallucination rates by 40%-60% across mainstream MLLM backbones while maintaining general multimodal capabilities. The framework offers an attention-based perspective on addressing MLLM hallucinations.

Key takeaway

For Machine Learning Engineers developing or deploying Multimodal Large Language Models, ADAPT offers a direct, attention-based solution to a critical hallucination problem. If you are struggling with MLLM outputs inconsistent with provided images, implementing ADAPT's cross-attention visual anchor, attention-supervised inference, and Visual Attention Guidance DPO can significantly reduce hallucination rates by 40-60%. This approach preserves general multimodal capabilities, providing a robust method to enhance MLLM faithfulness.

Key insights

MLLM hallucination stems from progressive text-to-image cross-attention degradation, which ADAPT mitigates by aligning attention dynamics.

Principles

• MLLM hallucination correlates with cross-attention degradation.
• Targeting attention dynamics improves visual grounding.
• Preference tuning can align responses to visual data.

Method

ADAPT refines a cross-attention visual anchor from early decoding, employs an attention-supervised inference mechanism to correct online drift, and uses Visual Attention Guidance DPO for preference alignment.

In practice

• Reduce MLLM hallucination rates by 40-60%.
• Improve MLLM visual grounding and faithfulness.
• Apply attention-based intervention for MLLM reliability.

Topics

• Multimodal LLMs
• Hallucination Mitigation
• Cross-Attention
• Preference Tuning
• Visual Grounding
• ADAPT Framework

Code references

• yao-ustc/ADAPT

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Reducing Hallucination in Vision-Language Models via Stage-wise Preference Optimization under Distribution Shift — Stage-wise preference optimization with targeted data construction effectively reduces VLM hallucination by focusing on difficult grounding boundaries.
• Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs — VIGIL mitigates MLLM visual laziness and hallucinations by causally grounding responses to visual input, penalizing "blind confidence" via counterfactual alignment.
• MACD: Model-Aware Contrastive Decoding via Counterfactual Data — MACD uses model feedback to create targeted counterfactual data, improving contrastive decoding for Video-LLM hallucination.
• When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs — LVLM hallucinations are largely driven by over-reliance on textual instruction priors, not just vision backbone limits.
• Causal Evidence for Attention Head Imbalance in Modality Conflict Hallucination — MLLM modality conflict arises from an imbalanced attention head routing structure, addressable by targeted intervention.
• Mind the Heads: Topological Representation Alignment for Multimodal LLMs — HeRA improves MLLMs by aligning individual attention heads based on topological structure, surprisingly focusing on the least aligned.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.