Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability

2026-05-23 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

The paper introduces Synergistic Faithfulness ("F_syn"), a novel metric for evaluating Vision-Language Model (VLM) explainability, addressing limitations of current unimodal perturbation metrics. Existing methods fail because multimodal datasets often contain language priors and modality biases, leading to cross-modal redundancy where VLMs can answer queries using text alone. This causes unimodal metrics to penalize faithful explainers and results in contradictory visual and textual rankings. "F_syn" is rooted in the Shapley Interaction Index, strictly isolating the joint Harsanyi dividend between modalities. It achieves a strong Spearman correlation ("rho=0.92") with exact Shapley Interaction Index while providing a "24x" computational speedup. Evaluating 8 XAI methods across 3 VLM architectures (LLaVA-1.5, Qwen2.5-VL, InternVL-3.5) and 3 benchmark datasets (RePOPE, CVBench, MMStar) revealed that attention-based explainers significantly outperform VLM-native approaches in capturing true cross-modal synergy.

Key takeaway

For AI Scientists and MLOps Engineers deploying Vision-Language Models in high-stakes environments, you must move beyond unimodal explainability metrics. Relying on visually plausible but mathematically unfaithful explanations risks deploying dangerously ungrounded models. Instead, prioritize evaluating XAI methods using synergistic metrics like "F_syn" to verify genuine cross-modal reasoning. Your current VLM-native explainers may over-index on visual salience; consider attention-based methods for more faithful insights into model behavior.

Key insights

Unimodal VLM explainability metrics fail due to cross-modal redundancy; synergistic metrics are crucial for faithful evaluation.

Principles

• Multimodal XAI needs cross-modal synergy.
• Unimodal metrics yield unstable rankings.
• Attention-based explainers capture synergy better.

Method

Synergistic Faithfulness ("F_syn") approximates the Shapley Interaction Index by evaluating the Harsanyi dividend across continuous perturbation trajectories, achieving a "24x" speedup over exact computation.

In practice

• Prioritize attention-based XAI methods like AttnLRP.
• Avoid VLM-native explainers for true synergy.
• Use "F_syn" to audit VLM reasoning.

Topics

• Vision-Language Models
• Explainable AI
• Cross-Modal Explainability
• Shapley Interaction Index
• Faithfulness Metrics
• Attention Mechanisms

Code references

• xmed-lab/TAM
• RuoyuChen10/EAGLE

Best for: Research Scientist, AI Scientist, MLOps Engineer, AI Ethicist

Related on AIssential

• Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability — New metric ℱ_{syn} rigorously evaluates VLM cross-modal synergy, addressing limitations of unimodal explainability metrics.
• Interpretable and Explainable Surrogate Modeling for Simulations: A State-of-the-Art Survey and Perspectives on Explainable AI for Decision-Making — Integrating XAI with surrogate models enhances transparency and diagnostic capabilities for complex simulations, bridging efficiency and interpretability.
• From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models — The survey unifies vision-language perception in MLLMs, proposing a five-stage evolution taxonomy and future directions for AGI.
• LoMo: Local Modality Substitution for Deeper Vision-Language Fusion — LoMo addresses VLM "carrier sensitivity" by creating semantically equivalent text and image representations through local modality substitution.
• Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement — Effective interleaved multimodal reasoning requires explicit structural supervision at modality boundaries, not just end-task optimization.
• Context Unrolling in Omni Models — Unified multimodal training enables "Context Unrolling" for enhanced cross-modal reasoning and performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.