The Compositional Grounding Gap: Why Vision-Language Models Fail at Relational Reasoning and How to Fix It

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Large vision-language models (LVLMs) exhibit a "compositional grounding gap," consistently failing at tasks requiring relational reasoning, such as distinguishing "the cat on the mat" from "the mat on the cat." This failure stems from vision-language alignment processes that rely on pooled, order-invariant visual features, which create "compositional blind spots" where distinct scenes map to identical representations. The number of these blind spots grows factorially with scene complexity, establishing a fundamental architectural limit. To address this, a new training-free, test-time method called REGROUND is proposed. REGROUND re-introduces spatial structure by employing relation-guided cross-attention over spatial visual tokens, directed by a lightweight text query parse. Applied to LLaVA-1.5, REGROUND improves compositional accuracy by +8.6 points on Winoground, +8.4 on ARO-Relation, +6.4 on SugarCrepe, and +8.4 on VSR, demonstrating consistent gains across various LVLMs.

Key takeaway

For AI Scientists and NLP Engineers developing or deploying vision-language models, recognize that current architectures often possess fundamental "compositional blind spots" due to pooled visual features. You should consider integrating methods like REGROUND to enhance relational reasoning without fine-tuning. This approach, which re-introduces spatial structure via relation-guided cross-attention, can significantly improve accuracy on benchmarks like Winoground and ARO-Relation, making your LVLMs more robust for complex scene understanding.

Key insights

Vision-language models fail relational reasoning due to pooled features creating "compositional blind spots" that REGROUND fixes.

Principles

• Pooled visual features create "compositional blind spots."
• Blind spots grow factorially with scene complexity.
• Spatial structure is crucial for relational grounding.

Method

REGROUND is a training-free, test-time method that re-introduces spatial structure via relation-guided cross-attention over spatial visual tokens, directed by a lightweight text query parse.

In practice

• Apply REGROUND to LLaVA-1.5 for +8.6 Winoground accuracy.
• Use parse guidance, token-level attention, and relation masking.

Topics

• Vision-Language Models
• Compositional Reasoning
• Relational Reasoning
• Pooled Visual Features
• REGROUND
• Cross-Attention
• LLaVA-1.5

Best for: Research Scientist, AI Scientist, NLP Engineer, Computer Vision Engineer

Related on AIssential

• Reinforcing Dual-Path Reasoning in Spatial Vision Language Models — SR-REAL improves spatial VLMs by combining linguistic deduction and 3D geometric grounding via reinforcement learning for robust, generalizable reasoning.
• Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning — Symbolic input significantly improves LLM performance on abstract visual reasoning, identifying representation as a key bottleneck.
• Why Far Looks Up: Probing Spatial Representation in Vision-Language Models — Vision-language models consistently conflate vertical image position with distance, hindering robust spatial reasoning.
• Thinking with Visual Grounding — Vision-language models enhance reasoning by explicitly grounding natural-language thoughts to specific image regions.
• Why Vision Language Models Ignore What They See [Munawar Hayat] - 758 — Current VLMs struggle with physics-based generation, visual grounding, and multi-person image fidelity, necessitating improved training and architectural approaches.
• Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning — Structured qualitative reasoning at inference time improves VLM robustness to visual illusions without fine-tuning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.