OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model

2026-04-22 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

OMIBench is a new benchmark designed to evaluate Olympiad-level reasoning in large vision-language models (LVLMs) when evidence is distributed across multiple images. Current benchmarks often focus on single-image analysis, overlooking multi-image contextual information. OMIBench includes problems from biology, chemistry, mathematics, and physics Olympiads, featuring manually annotated rationales and evaluation protocols for both exact and semantic answer matching. Extensive experiments reveal significant performance gaps in existing LVLMs, with even the most capable models like Gemini-3-Pro achieving only approximately 50% on the benchmark. This resource aims to facilitate research and development in multi-image reasoning capabilities for LVLMs.

Key takeaway

For AI Engineers developing or fine-tuning large vision-language models, OMIBench highlights a critical area for improvement: multi-image reasoning. Your models, even advanced ones like Gemini-3-Pro, likely perform at only 50% on these tasks. Prioritize developing architectures and training methodologies that can effectively integrate and reason over contextual information from multiple visual inputs to enhance real-world problem-solving capabilities.

Key insights

OMIBench evaluates LVLMs on Olympiad-level multi-image reasoning, revealing significant performance gaps.

Principles

• Multi-image context is crucial for advanced reasoning.
• Current LVLMs struggle with distributed visual evidence.

Method

OMIBench uses Olympiad problems from diverse subjects with manually annotated rationales and protocols for exact and semantic answer matching to evaluate multi-image reasoning.

In practice

• Test LVLMs on multi-image reasoning tasks.
• Identify specific weaknesses in contextual understanding.

Topics

• OMIBench
• Large Vision-Language Models
• Multi-Image Reasoning
• Olympiad-Level Reasoning
• Benchmark Evaluation

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields — OmniMatBench reveals significant multimodal reasoning gaps in MLLMs for materials science, with the best model scoring only 0.372.
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis — Current LVLMs struggle with fine-grained image recognition due to intertwined bottlenecks in visual representations, semantic grounding, and knowledge.
• 🗞️ Google releases Gemini 3 Deep Think, tops ARC-AGI 2 Benchmark With 84.6% — Advanced AI models are achieving human-level performance in complex reasoning and code generation through enhanced architectural and operational methods.
• CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences — CV-Arena offers a new benchmark and evaluation protocol for complex, instruction-guided image editing.
• Phi-4-reasoning-vision and the lessons of training a multimodal reasoning model — Efficient multimodal reasoning models can achieve high performance with less data and compute through careful design and data curation.
• Visual Reasoning Benchmark: Evaluating Multimodal LLMs on Classroom-Authentic Visual Problems from Primary Education — MLLMs struggle with dynamic visual reasoning despite proficiency in static visual tasks, posing risks for educational applications.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.