Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task

2026-07-01 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

A study investigates depth perception in Vision-Language Models (VLMs) to differentiate between pictorial depth cue understanding and language bias. Researchers combined depth-ordering and odd-one-out psychophysical tasks, presenting VLMs with images where one object's depth varied relative to identical others, requiring a "closer" or "farther" determination. Stimuli were generated from 3D scenes, allowing control over individual pictorial depth cues, while language effects were tested by varying referring expression clarity. A new metric was introduced to quantify vision-vs-language sensitivities. This methodology led to the creation of the Odd-One-Out Depth (O3-D) dataset, comprising 37K real and synthetic images and 147K image-question pairs. Evaluation of 12 VLMs on O3-D revealed depth-ordering accuracies between 47% and 56%, with no model performing above chance level, indicating under-utilization of depth cues. The study also identified a strong linguistic bias, and neither chain-of-thought nor in-context learning improved performance, suggesting static image data alone might be insufficient for robust depth understanding. All associated code and data are publicly available.

Key takeaway

For computer vision engineers developing VLMs for tasks requiring spatial understanding, recognize that current models exhibit significant linguistic bias and poor depth perception, with accuracies between 47% and 56%. Your VLM's performance in real-world 3D environments may be compromised by its under-utilization of pictorial depth cues. You should prioritize exploring training methodologies that incorporate more diverse visual data, potentially including 3D scene representations, rather than solely relying on static images or prompting techniques like CoT or ICL, to improve true depth understanding.

Key insights

VLMs struggle with depth perception, showing strong linguistic bias and under-utilizing pictorial cues, even with advanced prompting.

Principles

• VLMs exhibit significant linguistic bias in depth perception.
• Static image data alone may limit VLM depth understanding.
• Pictorial depth cues are under-utilized by current VLMs.

Method

The Odd-One-Out Depth (O3-D) methodology combines depth-ordering and odd-one-out tasks using controlled 3D scene stimuli and a novel metric to quantify vision-vs-language sensitivities in VLMs.

In practice

• Evaluate VLM depth perception using the O3-D dataset.
• Analyze VLM biases with the vision-vs-language sensitivity metric.
• Consider multimodal data beyond static images for depth tasks.

Topics

• Vision-Language Models
• Depth Perception
• Pictorial Cues
• Linguistic Bias
• O3-D Dataset
• Computer Vision

Code references

• lyiqian/o3-d

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Anchored, Not Graded: Vision-Language Models Fail at Slant-from-Texture Perception — VLMs fail to express continuous geometric perception due to a language readout bottleneck, despite encoding the information.
• DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues — MLLMs significantly underperform humans in high-resolution, multi-image fine-grained perception, highlighting a critical research gap.
• Disparities In Negation Understanding Across Languages In Vision-Language Models — VLMs exhibit affirmation bias, with negation understanding varying significantly across typologically diverse languages.
• LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models — LOCUS enhances MLLM fine-grained perception by internalizing local visual cue search via a training-time proxy task.
• LaViSA: A Language and Vision Structural Ambiguity Benchmark — VLMs struggle with visual disambiguation of structural ambiguity, especially subtle semantic distinctions.
• Diagnosing Visual Ignorance in Vision-Language Models — VLMs' visual ignorance results from internal routing failures and benchmarks that reward language-prior reliance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.