Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs

2026-05-13 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

A new benchmark, IMAVB, comprising 500 movie clips, evaluates omnimodal large language models' ability to detect textual claims that contradict their visual or auditory input. The benchmark employs a 2x2 design, varying target modality (vision, audio) and premise condition (standard, misleading), to isolate conflict detection from general multimodal comprehension. Across eight open-source omnimodal LLMs and Gemini 3.1 Pro, a "Representation-Action Gap" was identified: models' internal states encode premise-perception mismatches, but their outputs rarely reject false claims. Models exhibit under-rejection (answering misleading questions as if true) and over-rejection (rejecting standard questions), with audio grounding performing worse than vision. This gap persists despite seven prompt variations, suggesting a translation bottleneck rather than a perception failure. A probe-guided logit adjustment (PGLA) intervention improved rejection behavior.

Key takeaway

For AI Engineers developing omnimodal LLMs, understanding the Representation-Action Gap is crucial. Your models may internally detect contradictions but fail to act on them, leading to unreliable outputs. Focus development efforts on improving the translation layer between internal representations and output generation, potentially by integrating techniques like probe-guided logit adjustment, to enhance grounding and prevent misleading responses.

Key insights

Omnimodal LLMs struggle to reject textual claims contradicting their sensory input, despite internal encoding of mismatches.

Principles

• Grounding failures are often translation, not perception.
• Audio grounding is more challenging than vision grounding.

Method

The IMAVB benchmark uses a 2x2 design (modality x premise condition) with long-form movie clips to measure conflict detection in omnimodal LLMs, distinguishing it from general comprehension.

In practice

• Use IMAVB to benchmark omnimodal LLM grounding.
• Implement PGLA for improved rejection behavior.

Topics

• Omnimodal LLMs
• Representation-Action Gap
• IMAVB Benchmark
• Multimodal Grounding
• Conflict Detection

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models — MLLMs struggle to translate visual perception into context-specific actions, revealing a significant "perception-to-action gap."
• Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement — Explicitly supervising modality transitions in interleaved thinking prevents "Modal Isolation," enhancing cross-modal coherence and task accuracy in multimodal models.
• Nobody gets this right — The distinction between language models and world models is diminishing as AI becomes multimodal and adept at abstract mathematical representations.
• When Vision Speaks for Sound — MLLMs often hallucinate audio from visual cues, exhibiting a "Clever Hans effect" instead of true audio-visual grounding.
• MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue — Decoupling linguistic disambiguation from visual localization significantly improves multimodal grounding in dynamic dialogue.
• DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models — Audio MLLMs exhibit pervasive text dominance, prioritizing textual cues over acoustic signals, even when perceiving acoustic variations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.