When Image and Text Disagree: Cross-Modal Evidence Conflict in Multimodal Retrieval-Augmented Generation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, short

Summary

The Cross-Modal Conflict Benchmark (CMC-Bench) evaluates how multimodal retrieval-augmented generation (RAG) systems manage contradictory evidence from retrieved text and images. Utilizing 3,768 instances from ChartQA and MMMU evaluation splits, the study benchmarked four open vision-language models (VLMs) across four conflict types: factual, temporal, entity, and granularity. It also examined four evidence conditions, including aligned, image-correct, text-correct, and both-wrong. Key findings indicate that cross-modal disagreement significantly degrades performance, with accuracy changes ranging from 0.17 to 0.46 relative to aligned evidence. Models often exhibit a modality lean instead of robust arbitration, making text-leaning systems particularly vulnerable when only the image provides correct information. The research also highlights that combining abstention and fabrication into a single hallucination score obscures crucial behavioral differences, noting Qwen3-VL-4B abstains on 31.7% of conflicts while Gemma-3n-E2B fabricates answers in 51.9% of conflicts.

Key takeaway

For Machine Learning Engineers developing or evaluating multimodal RAG systems, you must explicitly account for cross-modal evidence conflicts. Your evaluation metrics should distinguish between model abstention and fabrication, as merging these behaviors obscures critical reliability issues. Prioritize developing arbitration mechanisms that prevent modality lean, especially when one modality is solely correct, to improve system robustness and reduce performance degradation by up to 0.46 in accuracy.

Key insights

Multimodal RAG systems struggle with cross-modal evidence conflicts, often exhibiting modality bias and requiring distinct evaluation of abstention versus fabrication.

Principles

• Cross-modal disagreement severely degrades VLM performance.
• Models often lean on one modality, not arbitrate reliably.
• Hallucination scores should separate abstention from fabrication.

Method

The CMC-Bench evaluates multimodal RAG by testing VLMs against 3,768 instances with four conflict types and four evidence conditions (aligned, image-correct, text-correct, both-wrong) to measure performance degradation and modality lean.

In practice

• Benchmark VLMs using CMC-Bench for conflict handling.
• Analyze modality lean in RAG system responses.
• Distinguish abstention from fabrication in VLM metrics.

Topics

• Multimodal RAG
• Cross-Modal Conflict Benchmark
• Vision-Language Models
• Hallucination Detection
• Evidence Arbitration
• Qwen3-VL-4B
• Gemma-3n-E2B

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation — Multilingual RAG systems face significant challenges with contradictory evidence, requiring specialized diagnostic tools and pipelines like X-MADAM-RAG.
• MODE-RAG: Manifold Outlier Diagnosis and Energy-based Retrieval-Augmented Generation Evaluation — MODE-RAG dynamically gates M-RAG interventions using Variational Free Energy and attention states to reduce cross-modal hallucinations and fabrications.
• Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated — Benchmarking VLMs for urban perception must account for human disagreement and abstention, treating them as valid measurement outcomes.
• The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models — Language representations in MLMs systematically overshadow vision, causing underperformance in visual tasks.
• Benchmarking RAG Architectures Locally on a Real Financial PDF — Part 3: The Measurement Problem — Multimodal RAG evaluation requires answer correctness against independent references, as text-grounded metrics misrepresent performance.
• ConflictRAG: Detecting and Resolving Knowledge Conflicts in Retrieval Augmented Generation — ConflictRAG systematically detects, classifies, and resolves knowledge conflicts in RAG systems, significantly improving answer correctness and reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.