Egocentric Bias in Vision-Language Models

2026-02-19 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A new diagnostic benchmark, FlipSet, evaluates Level-2 Visual Perspective Taking (L2 VPT) in 103 Vision-Language Models (VLMs), revealing a systematic "egocentric bias." The benchmark requires models to simulate a 180-degree rotation of 2D character strings from another agent's viewpoint, isolating spatial transformation from 3D scene complexity. Results show that 91.3% of VLMs perform below the 25% chance level, with an average accuracy of 8.96%. A striking 75.88% of errors are egocentric, meaning models reproduce the camera's viewpoint. Control experiments on 24 models further expose a compositional deficit: models achieve high Theory-of-Mind (ToM) accuracy (90.4%) and above-chance mental rotation (26.1%) in isolation, but catastrophically fail L2 VPT (10.3%) when integration of these abilities is required. This indicates VLMs lack mechanisms to bind social awareness to spatial operations.

Key takeaway

For Computer Vision Engineers developing multimodal AI, this research highlights a critical limitation: current VLMs struggle with Level-2 visual perspective taking due to an inability to integrate social awareness with spatial reasoning. You should prioritize architectural innovations that support model-based spatial reasoning and targeted training on perspective-invariant representations, rather than solely relying on scaling existing pattern-matching mechanisms, to achieve more human-like situated social intelligence.

Key insights

VLMs exhibit a profound egocentric bias, failing to integrate social awareness with spatial transformation for Level-2 visual perspective taking.

Principles

• L2 VPT requires both theory of mind and mental rotation.
• Egocentric bias dominates VLM errors in perspective taking.
• Compositional deficits hinder VLM integration of cognitive abilities.

Method

FlipSet uses 2D character string rotation tasks and control conditions to isolate Theory of Mind, Mental Rotation, and L2 VPT, evaluating 103 VLMs under zero-shot conditions with diagnostic error analysis.

In practice

• Target VLM training on multi-view or egocentric-to-allocentric data.
• Develop systems for model-based spatial simulation.
• Explore architectures for fine-grained visual encoding.

Topics

• Visual Perspective Taking
• Vision-Language Models
• Egocentric Bias
• Mental Rotation
• Cognitive Benchmarking

Best for: Computer Vision Engineer, Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs — Current MLLMs struggle with visual cognitive and visuospatial reasoning compared to humans.
• Uneven Evolution of Cognition Across Generations of Generative AI Models — Generative AI models exhibit uneven cognitive development, excelling in language but struggling with visual-perceptual reasoning.
• From Hallucination to Grounding: Diagnosing Visual Spatial Intelligence via CRISP — CRISP evaluates VLM visual spatial intelligence by decoupling latent reasoning from perceptual bottlenecks using 3D Scene Graphs.
• Seeing Together:Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models — Multi-robot egocentric spatial reasoning benefits from physics-informed MLLMs and specialized cooperative datasets.
• Why Far Looks Up: Probing Spatial Representation in Vision-Language Models — Vision-language models consistently conflate vertical image position with distance, hindering robust spatial reasoning.
• Visual language models show widespread visual deficits on neuropsychological tests — VLMs excel at high-level object recognition but fail at foundational low- and mid-level visual concepts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.