Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems

2026-04-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

A new benchmark, MM-AQA, has been introduced to evaluate effective abstention (EA) in multimodal reasoning systems, specifically vision-language models (VLMs) and multi-agent systems (MAS). MM-AQA creates unanswerable instances by transforming answerable ones based on visual modality dependency and evidence sufficiency. Evaluating three frontier VLMs (both closed and open-source) and two MAS architectures across 2079 samples revealed several key findings. Under standard prompting, VLMs rarely abstain, with simple confidence baselines outperforming this setup. While MAS improves abstention, it introduces an accuracy-abstention trade-off. Sequential MAS designs performed comparably to or better than iterative variants, suggesting miscalibration, not reasoning depth, as the primary bottleneck. Models tend to abstain when evidence is entirely absent but attempt reconciliation with degraded or contradictory evidence.

Key takeaway

For research scientists developing multimodal AI, you should prioritize integrating abstention-aware training into your model development pipeline. Relying solely on advanced prompting or increasing agent count will not yield robust abstention capabilities, as miscalibration is a significant barrier. Focus on explicit training for recognizing evidence insufficiency to enhance system reliability and trustworthiness.

Key insights

Effective abstention in multimodal systems requires abstention-aware training, not just better prompting or more agents.

Principles

• VLMs rarely abstain under standard prompting.
• MAS improves abstention but creates a trade-off.
• Miscalibration, not reasoning depth, limits abstention.

Method

MM-AQA constructs unanswerable multimodal instances by transforming answerable ones along axes of visual modality dependency and evidence sufficiency to evaluate abstention capabilities.

In practice

• Implement abstention-aware training for VLMs.
• Use confidence baselines for initial abstention.
• Prioritize sequential MAS designs over iterative.

Topics

• Multimodal Reasoning Systems
• Effective Abstention
• Vision-Language Models
• Multi-Agent Systems
• MM-AQA Benchmark

Best for: Research Scientist, AI Scientist, NLP Engineer, Computer Vision Engineer

Related on AIssential

• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs — Visual latents in MLLMs can be "silenced" by training objectives, but their reasoning capacity can be unleashed at inference time.
• Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement — Interleaved thinking in multimodal models fails due to "Modal Isolation," requiring explicit transition-level supervision for coherence.
• M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions — M$^3$Exam benchmarks multimodal memory for realistic user-agent interactions, revealing key gaps and proposing M$^3$Proctor for efficiency.
• CORE: Conflict-Oriented Reasoning for General Multimodal Manipulation Detection — Multimodal misinformation's essence lies in intrinsic conflicts, which CORE detects by enhancing MLLMs for generalizable manipulation detection.
• MAny: Merge Anything for Multimodal Continual Instruction Tuning — MAny mitigates dual-forgetting in MLLMs via training-free merging of perceptual and reasoning knowledge.
• Social Norm Reasoning in Multimodal Language Models: An Evaluation — MLLMs demonstrate stronger social norm reasoning in text than images, with GPT-4o leading performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.