From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

MazeBench, a new benchmark comprising 110 procedurally generated maze images across nine controlled groups, evaluates how multimodal models solve visual spatial tasks. While GPT-5.4 achieved 91% and Gemini 3.1 Pro 79%, analysis of 16 model configurations from OpenAI, Anthropic, Google, and Alibaba revealed these scores are misleading. Models translate images into text grids and brute-force paths via serial enumeration, consuming 1,710–22,818 tokens per solve. Without added reasoning budgets, all configurations scored only 2–12%, and 20x20 ultra-hard mazes caused token limits. Qualitative analysis confirmed a universal two-stage strategy: image-to-grid translation followed by natural language path search, essentially BFS in prose. Claude's performance improved from 6% to 80% when given a text grid directly, highlighting vision quality as a bottleneck for weaker models. Models consistently ignored instructions to "reason visually," defaulting to grid enumeration.

Key takeaway

For AI Scientists and Machine Learning Engineers evaluating multimodal models for spatial reasoning, understand that high accuracy on tasks like mazes often indicates brute-force token-level search, not genuine visual planning. You should design benchmarks that prevent simple grid enumeration or explicitly test for visual reasoning capabilities, as models tend to ignore instructions to "reason visually" and revert to their default two-stage strategy. This insight is crucial for selecting models truly capable of complex spatial problem-solving.

Key insights

Multimodal models achieve high maze accuracy by brute-forcing solutions via text-grid translation and serial enumeration, not genuine visual planning.

Principles

• High benchmark scores can mask brute-force token-level search.
• Vision quality often bottlenecks weaker multimodal model performance.
• Models default to grid enumeration despite "visual reasoning" instructions.

Method

Models universally employ a two-stage strategy: image-to-grid translation, followed by step-by-step path search in natural language, effectively implementing BFS in prose.

In practice

• Use diagnostic benchmarks to reveal underlying model strategies.
• Test models with varying reasoning budgets and grid complexities.
• Evaluate vision quality separately from reasoning ability.

Topics

• MazeBench
• Multimodal Models
• Visual Spatial Reasoning
• Brute-Force Search
• Large Language Models
• Model Evaluation

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning — Gains in latent visual reasoning stem from boundary markers and format, not visual memory encoded in latent slots.
• Can you *really* train AI to "get" videos just by showing it a million of them? — Current video models excel at generation but lack systematic evaluation for genuine spatiotemporal reasoning.
• MathVis-Fine: Aligning Visual Supervision with Necessity via Progressive Dependency-Guided Training for Multimodal Mathematical Reasoning — MathVis-Fine improves multimodal mathematical reasoning by aligning visual supervision with necessity via progressive dependency-guided training.
• VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis — VGenST-Bench shifts spatio-temporal reasoning evaluation from passive curation to active, controlled video synthesis for MLLMs.
• Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs — Chain-of-Thought prompting degrades visual spatial reasoning in multimodal LLMs due to shortcut learning and hallucination.
• See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch — ScratchWorld benchmarks multimodal GUI agents in Scratch, revealing a gap between planning and fine-grained GUI manipulation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.