Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis

2026-06-17 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

A new benchmark, FG-BMK, has been introduced to evaluate Large Vision-Language Models (LVLMs) on fine-grained image tasks, an area previously insufficiently understood. FG-BMK comprises 1.01 million questions and 0.28 million images, spanning common object-centric and specialized domains. It assesses LVLMs through both dialogue-level fine-grained semantic recognition and feature-level visual discriminability, utilizing human-oriented and machine-oriented paradigms. This diagnostic approach helps pinpoint whether LVLM failures stem from inadequate visual representations, weak visual-to-semantic grounding, or limited fine-grained knowledge. Extensive experiments on various LVLMs reveal that current models are insufficient fine-grained recognizers, encountering bottlenecks in visual representations, semantic grounding, modality alignment, and category-level knowledge. These findings offer crucial guidance for future data construction and model design to develop more reliable LVLMs for fine-grained visual tasks. The code is open-source.

Key takeaway

For Machine Learning Engineers developing or deploying Large Vision-Language Models for fine-grained visual tasks, you should recognize that current LVLMs exhibit significant limitations. Your model designs must specifically address bottlenecks in visual representations, semantic grounding, modality alignment, and category-level knowledge. Utilize diagnostic benchmarks like FG-BMK to pinpoint specific failure modes and guide your data construction and model architecture choices for improved reliability.

Key insights

Current LVLMs struggle with fine-grained image recognition due to intertwined bottlenecks in visual representations, semantic grounding, and knowledge.

Principles

• Fine-grained visual tasks require robust semantic grounding.
• LVLM failures often involve multiple intertwined bottlenecks.
• Diagnostic benchmarks are crucial for model improvement.

Method

FG-BMK evaluates LVLMs on fine-grained tasks by jointly assessing dialogue-level semantic recognition and feature-level visual discriminability using human- and machine-oriented paradigms for diagnostic analysis.

In practice

• Use FG-BMK to diagnose LVLM fine-grained limitations.
• Focus data construction on improving semantic grounding.
• Design models to address modality alignment issues.

Topics

• Large Vision-Language Models
• Fine-Grained Recognition
• LVLM Benchmarking
• Semantic Grounding
• Modality Alignment

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models — Current Video-Language Models exhibit fragile temporal and visual grounding, often relying on linguistic priors over actual video evidence.
• Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games — MLLMs struggle with reconstructing and acting on past, hidden observations in non-Markovian games.
• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models — Biomedical VLMs exhibit pervasive perceptual weaknesses, especially in object detection and open-ended tasks, despite medical adaptation.
• Diagnosing Visual Ignorance in Vision-Language Models — Vision-Language Models frequently prioritize language priors over visual evidence due to internal routing failures and flawed benchmark evaluations.
• NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models — NextMotionQA is a new benchmark for human motion understanding, revealing VLM limitations in fine-grained analysis.
• Scaling Capability in Token Space: An Analysis of Large Vision Language Model — LVLMs exhibit predictable sublinear and linear scaling regimes based on vision token count.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.