HoloCount: A Holistic Visual Counting Benchmark for MLLMs

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

HoloCount is a new, comprehensive benchmark designed to rigorously evaluate Multimodal Large Language Models' (MLLMs) visual counting capabilities, specifically addressing their persistent numerical hallucinations. Developed by Meituan, it features a three-level hierarchical taxonomy: Semantic Counting, Analytical Counting, and Robustness Testing. The benchmark comprises a meticulously curated dataset of 2,480 QA pairs across 20 fine-grained tasks and 1,481 unique visual concepts. An exhaustive evaluation of over 20 state-of-the-art MLLMs revealed a critical performance gap, with models degrading significantly as tasks transitioned from basic perception to complex analytical reasoning and adverse scenarios like high-density scenes or linguistic biases. HoloCount provides a systematic landscape of current MLLM counting limitations and offers a roadmap for developing more grounded and reliable multimodal systems.

Key takeaway

For AI Scientists and Machine Learning Engineers developing MLLMs, HoloCount highlights critical weaknesses in quantitative precision and reasoning. You should prioritize architectural improvements that enhance spatial reasoning, set-based logic, and robustness against high-density scenes or linguistic biases. Consider integrating explicit "thinking mode" mechanisms to improve counting accuracy, especially for complex analytical tasks, and focus on vision-first arbitration to prevent numerical hallucinations when objects are absent or priors conflict with visual evidence.

Key insights

MLLMs struggle with quantitative precision, especially in complex visual counting tasks requiring reasoning and robustness.

Principles

• MLLM counting performance degrades significantly from basic perception to analytical reasoning and robustness challenges.
• Smaller MLLMs can surprisingly outperform advanced proprietary models on null-target counting tasks.
• "Thinking mode" consistently improves MLLM counting accuracy across various model scales.

Method

HoloCount uses a three-level hierarchical taxonomy (Semantic, Analytical, Robustness) with 20 fine-grained tasks and 2,480 QA pairs, curated via MLLM generation and rigorous human verification.

In practice

• Implement "thinking mode" for MLLMs to enhance counting accuracy in complex scenarios.
• Prioritize vision-first arbitration mechanisms to counter linguistic priors in MLLM counting.
• Improve fine-grained attribute filtering to prevent over-counting of visual distractors.

Topics

• Visual Counting
• MLLM Benchmarking
• Numerical Hallucinations
• Spatial Reasoning
• Robustness Testing
• Multimodal AI

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models — MLLMs struggle to translate visual perception into context-specific actions, revealing a significant "perception-to-action gap."
• Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMs — KidGym evaluates MLLM cognitive abilities using a 2D grid-based benchmark inspired by child intelligence tests.
• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs — Visual latents in MLLMs can be "silenced" by training objectives, but their reasoning capacity can be unleashed at inference time.
• CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmark — Advancing MLLM spatial intelligence requires large-scale data, systematic evaluation, and explicit cross-view object alignment.
• Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games — MLLMs struggle with reconstructing and acting on past, hidden observations in non-Markovian games.
• Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference — Retrieval-augmented reliability-aware inference significantly reduces visual hallucinations in MLLMs without retraining.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.