ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

2025-01-16 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

ZeroBench is a new, lightweight visual reasoning benchmark designed to expose significant shortfalls in contemporary Large Multimodal Models (LMMs) regarding image interpretation and spatial cognition. Comprising 100 manually curated main questions and 334 subquestions, ZeroBench is intentionally difficult, with all 20 evaluated frontier LMMs, including o1 pro, Gemini 2 Flash Thinking, and Claude 3.5 Sonnet v2, scoring 0.0% pass@1 on the main questions. While some models achieved low non-zero scores on pass@5 (Gemini 2 Flash Thinking at 7%), none showed consistent correct answers across multiple samplings. Subquestions, however, successfully differentiated model performance, with Claude 3.5 Sonnet v2 achieving 24.30% pass@1. Error analysis revealed that LMM failures are predominantly due to visual interpretation issues, such as counting and understanding spatial relations, rather than logical reasoning. ZeroBench is publicly available to encourage progress in visual understanding.

Key takeaway

For AI Scientists and Machine Learning Engineers developing or evaluating Large Multimodal Models, ZeroBench highlights a critical need to re-prioritize foundational visual perception. Your current LMMs likely struggle with basic tasks like counting and spatial reasoning, even if they perform well on existing benchmarks. You should integrate ZeroBench into your evaluation suite and focus development efforts on enhancing low-level visual interpretation rather than solely scaling reasoning capabilities, which current models already over-emphasize.

Key insights

Large Multimodal Models fundamentally struggle with visual interpretation and spatial reasoning, failing a new "impossible" benchmark.

Principles

• Benchmarks must be difficult enough to maintain relevance against rapid LMM advancements.
• LMMs' primary failure mode in visual tasks is interpretation, not complex reasoning.
• Lightweight benchmarks are essential for cost-effective evaluation of models using test-time compute.

Method

A 4-part pipeline curated 100 questions: iterative feedback, initial LMM evaluation, thorough review for difficulty and broad answer space, and adversarial filtering of any correctly answered questions. Subquestions were added for differentiation.

In practice

• Integrate ZeroBench into LMM evaluation pipelines to test advanced visual reasoning.
• Prioritize research into LMMs' low-level visual perception capabilities.
• Consider cost implications of test-time compute scaling for benchmark evaluations.

Topics

• ZeroBench
• Large Multimodal Models
• Visual Reasoning
• Benchmark Evaluation
• Spatial Cognition
• Visual Interpretation

Code references

• gkamradt/LLMTest_NeedleInAHaystack
• open-compass/opencompass

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark — Visual diversity is crucial for robust multimodal reasoning benchmark development.
• The mirage of visual understanding in current frontier models — Current LLMs can generate convincing visual descriptions and reasoning without actual image input, termed "mirage reasoning."
• From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning — Multimodal models achieve high maze accuracy by brute-forcing solutions via text-grid translation and serial enumeration, not genuine visual planning.
• OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields — OmniMatBench reveals significant multimodal reasoning gaps in MLLMs for materials science, with the best model scoring only 0.372.
• DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues — MLLMs significantly underperform humans in high-resolution, multi-image fine-grained perception, highlighting a critical research gap.
• Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly — Current LVLMs significantly struggle with fine-grained spatio-temporal reasoning and object interaction in complex video scenarios.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.