OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents

2026-07-10 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Computer Vision · Depth: Expert, quick

Summary

OmniMapBench is a new benchmark designed to evaluate visual-centric reasoning in Large Vision-Language Models (LVLMs) on diverse map documents. It addresses a critical limitation in existing document understanding benchmarks where visual content is often reducible to text, allowing high performance without genuine visual grounding. Comprising 2,096 manually annotated question-answer pairs across 1,603 map documents from nine categories, OmniMapBench probes a hierarchy of skills from perception to multi-step visual reasoning. The benchmark introduces the Visual Dependency Index (VDI), a metric quantifying accuracy drop when images are replaced with text, confirming its focus on irreducible visual reasoning. Evaluations of 25 leading LVLMs revealed a significant performance gap, with the top model achieving only 75.03% accuracy, highlighting the benchmark's challenge.

Key takeaway

For AI scientists and machine learning engineers developing or evaluating Large Vision-Language Models for document understanding, you should integrate OmniMapBench into your evaluation suite. This benchmark exposes current LVLM weaknesses in genuine visual-centric reasoning on map documents, with top models reaching only 75.03% accuracy. Using OmniMapBench will help you identify and address critical gaps in your models' ability to process irreducible visual information, driving progress beyond text-reducible approaches.

Key insights

OmniMapBench provides a robust benchmark for evaluating visual-centric reasoning in LVLMs on map documents, addressing limitations of text-reducible visual content.

Principles

• Many document understanding benchmarks allow text-reducible visual content.
• Higher Visual Dependency Index (VDI) indicates stronger irreducible visual reasoning.

Method

The Visual Dependency Index (VDI) quantifies visual dependency as the accuracy drop when images are replaced with question-agnostic descriptions.

In practice

• Evaluate LVLMs on OmniMapBench to assess genuine visual reasoning.
• Utilize VDI to measure a model's reliance on visual input.

Topics

• OmniMapBench
• LVLMs
• Visual Reasoning
• Map Documents
• Document Understanding
• Benchmarking
• Visual Dependency Index

Code references

• SIGMME/OmniMapBench

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark — Visual diversity is crucial for robust multimodal reasoning benchmark development.
• OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning — OmniTraffic is a benchmark and pipeline for spatio-temporal traffic reasoning, revealing significant MLLM gaps in complex traffic understanding.
• VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis — VGenST-Bench shifts spatio-temporal reasoning evaluation from passive curation to active, controlled video synthesis for MLLMs.
• OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice — VLMs exhibit a "Semantic-Physical Gap," failing catastrophically in food mass estimation and safety-critical health advice despite visual recognition accuracy.
• MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models — Current Omni LLMs lack robust cross-modal reasoning for safety, struggling to integrate diverse sensory cues effectively.
• MV-Bench: Benchmarking Multimodal Large Language Models for Coordinated Multi-View Interface Construction — MLLMs reproduce visual layouts well but fundamentally struggle with data semantics and interactive logic in multi-view interfaces.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.