Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming

2026-05-22 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Medical Imaging AI · Depth: Expert, long

Summary

The Look-Closer-Then-Diagnose framework significantly advances confidence-aware Ultrasound Visual Question Answering (VQA) by addressing current Vision-Language Model (VLM) limitations in ultrasound. It introduces a structured Zoom-then-Diagnose paradigm, which replicates sonographers' interactive search process for lesion-focused reasoning. Furthermore, within the Group Relative Policy Optimization (GRPO) framework, it integrates an uncertainty-aware reward derived from stochastic group-wise rollouts to estimate prediction consistency as a proxy for model confidence. Experiments across liver, breast, and thyroid datasets demonstrate a 39.3% improvement in lesion localization. The framework achieves superior diagnostic accuracy and alignment with sonographer preferences, maintaining lower Expected Calibration Error (e.g., 0.09 on Breast, 0.14 on Thyroid) and a high positive Entropy Gap (e.g., 0.28 on Breast, 0.34 on Thyroid), indicating better uncertainty reflection.

Key takeaway

For AI Scientists and Machine Learning Engineers developing medical imaging VLMs, particularly for ultrasound, you should integrate interactive, lesion-centric reasoning and uncertainty-aware training. This approach, mimicking sonographer workflow, significantly improves diagnostic accuracy and ensures your models express calibrated confidence, rather than overconfidence, in ambiguous cases. Consider building datasets that explicitly capture clinical consensus and disagreement to train for this crucial uncertainty alignment.

Key insights

A novel VLM framework integrates lesion-focused reasoning and uncertainty-aware rewards for improved ultrasound diagnosis.

Principles

• Sonographer workflow can inspire VLM design for medical tasks.
• Model confidence should align with clinical ambiguity and inter-observer variability.
• Group consistency can serve as a proxy for diagnostic certainty in stochastic models.

Method

The framework uses a Zoom-then-Diagnose paradigm for lesion localization, followed by diagnosis. It applies GRPO with an uncertainty-aware reward derived from group-wise rollout consistency to align model confidence with sonographer consensus.

In practice

• Apply Zoom-then-Diagnose for fine-grained visual tasks requiring localized attention.
• Use group consistency metrics for uncertainty calibration in VLMs operating in subjective domains.
• Develop datasets with explicit confidence labels to train models for uncertainty alignment.

Topics

• Ultrasound VQA
• Vision-Language Models
• Active Zooming
• Uncertainty-Aware AI
• Reinforcement Learning
• Medical Imaging Diagnosis

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Enhancing Pathological VLMs with Cross-scale Reasoning — Explicit cross-scale reasoning training and a leakage-aware VQA benchmark enhance pathological VLM understanding across magnifications.
• Implementing trust in non-small cell lung cancer diagnosis with a conformalized uncertainty-aware AI framework — TRUECAM enhances pathology AI trustworthiness by integrating uncertainty quantification, data filtering, and error rate control.
• DreamReg: Belief-Driven World Model for 2D-3D Ultrasound Registration — DreamReg uses a belief-driven world model and internal imagination for robust 2D-3D ultrasound registration, overcoming real-time challenges.
• Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support — LLMs exhibit a substantial performance gap in active, multi-turn clinical diagnosis compared to static, full-context evaluations.
• How AI Ultrasound Systems Are Improving Real-Time Diagnostic Accuracy in Clinical Settings — AI ultrasound systems enhance diagnostic accuracy and expand access by automating measurements, guiding image acquisition, and accelerating interpretation.
• Intelligent Healthcare Imaging Platform: A VLM-Based Framework for Automated Medical Image Analysis and Clinical Report Generation — A VLM-based framework automates medical image analysis and clinical report generation across multiple imaging modalities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.