BabyVision: Visual Reasoning Beyond Language

2025-01-09 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

The BabyVision benchmark evaluates Multimodal Large Language Models' (MLLMs) core visual abilities, independent of linguistic knowledge. It comprises 388 questions across 22 subclasses in four categories: Fine-grained Discrimination, Visual Tracking, Spatial Perception, and Visual Pattern Recognition. Empirical results show leading MLLMs significantly underperform humans; Gemini3-Pro-Preview scores 49.7%, far below the adult human average of 94.1% and lagging 6-year-olds by approximately 20 points. This reveals MLLMs lack fundamental visual primitives despite excelling in knowledge-heavy tasks. The paper also introduces BabyVision-Gen, a generative counterpart for visual output evaluation, and explores Reinforcement Learning with Verifiable Rewards (RLVR) for improving visual reasoning.

Key takeaway

For AI scientists and machine learning engineers developing multimodal models, recognize that current MLLMs possess a significant "verbalization bottleneck" that hinders foundational visual reasoning. You should prioritize architectural innovations that preserve visual fidelity throughout the reasoning process, rather than relying on linguistic compression. Consider integrating visual generation capabilities to allow models to "think" and respond visually, potentially bypassing this limitation and achieving more robust human-like perception.

Key insights

MLLMs lack fundamental visual primitives, struggling with basic tasks solvable by young children due to a "verbalization bottleneck."

Principles

• Visual understanding precedes language in humans.
• MLLMs rely on linguistic priors for vision.
• Verbalization bottlenecks degrade visual fidelity.

Method

BabyVision systematically assesses core visual abilities using 388 questions across 22 subclasses in four categories, minimizing linguistic reliance. BabyVision-Gen evaluates visual reasoning via image generation with an automatic evaluation toolkit.

In practice

• Test MLLMs on BabyVision to identify foundational visual gaps.
• Explore visual generation models for non-verbal reasoning tasks.
• Apply RLVR fine-tuning to enhance visual reasoning in open models.

Topics

• Multimodal LLMs
• Visual Reasoning
• BabyVision Benchmark
• Generative Models
• Early Vision
• Reinforcement Learning

Code references

• UniPat-AI/BabyVision
• hiyouga/EasyR1

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning — Gains in latent visual reasoning stem from boundary markers and format, not visual memory encoded in latent slots.
• Watch, Remember, Reason: Human-View Video Understanding with MLLMs — The human-view perspective unifies MLLM video understanding through watching, remembering, and reasoning capabilities.
• Why Vision Language Models Ignore What They See [Munawar Hayat] - 758 — Current VLMs struggle with physics-based generation, visual grounding, and multi-person image fidelity, necessitating improved training and architectural approaches.
• Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning — Symbolic input significantly improves LLM performance on abstract visual reasoning, identifying representation as a key bottleneck.
• Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs — Chain-of-Thought prompting degrades visual spatial reasoning in multimodal LLMs due to shortcut learning and hallucination.
• What is Holding Back Latent Visual Reasoning? — Latent tokens in VLMs currently offer minimal causal impact due to dataset limitations and poor inference-time generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.