SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation

2026-06-29 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Medical Imaging AI · Depth: Expert, medium

Summary

SHOVIR is a new benchmark designed to evaluate "vision shortcut" behavior in Vision-Language Models (VLMs) used for Radiology Report Generation (RRG). Current evaluation metrics for RRG often fail to assess if diagnostic statements are genuinely derived from visible pathological evidence, allowing models to exploit learned priors or spurious correlations. SHOVIR addresses this by extending two chest X-ray datasets, MIMIC-CXR and PadChest-GR, with per-box CheXpert labels. It employs image-level and disease-level occlusion experiments, comparing VLM predictions on clean images against those with localized, region-specific perturbations. This methodology identifies two failure modes: direct shortcuts, where a finding persists despite its visual evidence being removed, and contextual shortcuts, where detection degrades when co-occurring pathologies are occluded. Benchmarking eight state-of-the-art VLMs showed that high report quality does not guarantee strong spatial grounding, revealing a critical blind spot in current RRG evaluation protocols.

Key takeaway

For AI Scientists and Machine Learning Engineers developing or evaluating Radiology Report Generation models, you must move beyond traditional report-level metrics. Your evaluation protocols should incorporate region-aware assessments like SHOVIR's occlusion experiments to detect "vision shortcuts." This ensures your models genuinely ground diagnostic statements in visual evidence, preventing clinically fluent but unreliable outputs. Prioritize spatial grounding alongside lexical quality to build robust and trustworthy medical AI.

Key insights

Current VLM evaluation for radiology reports misses "vision shortcuts" where models rely on priors, not visual evidence.

Principles

• Lexical overlap metrics are insufficient for RRG.
• Clinical fluency can mask shallow visual grounding.
• Region-aware assessment is crucial for RRG.

Method

SHOVIR extends MIMIC-CXR and PadChest-GR with per-box CheXpert labels. It uses image-level and disease-level occlusion experiments to identify direct and contextual vision shortcuts.

In practice

• Test RRG models with localized image perturbations.
• Evaluate spatial grounding beyond report quality.
• Develop region-aware RRG evaluation protocols.

Topics

• Radiology Report Generation
• Vision-Language Models
• Vision Shortcut Learning
• SHOVIR Benchmark
• Medical AI Evaluation
• Chest X-ray Imaging

Code references

• KurtLabUW/BTReport

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology — Spatially grounded radiology VLMs can be trained at scale without manual annotations, maintaining language quality.
• UniRG: Scaling medical imaging report generation with multimodal reinforcement learning — Reinforcement learning with clinically meaningful rewards significantly improves medical vision-language model reliability and generality.
• Vision-language models for chest radiography do not always need the image — Many medical vision-language models for chest radiography achieve high accuracy by exploiting text-based finding-name priors rather than actual image interpretation.
• Rethinking Patient Education as Multi-turn Multi-modal Interaction — MedImageEdu benchmarks multi-turn, multimodal patient education, revealing gaps in visual grounding, safety, and handling emotional interactions.
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports — LLM evaluators for radiology reports suffer "discrimination bias," but targeted data synthesis and fine-tuning can sharpen clinical significance boundaries.
• E-MRL: Cross-view Aligned Evidence-driven Multimodal Reinforcement Learning for Reliable 3D Tumor Analysis — Grounding Vision-Language Models in specific visual evidence via reinforcement learning significantly reduces hallucinations in medical report generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.