Toward Training-Free Zero-Shot Anomaly Detection in 3D Medical Images: A Batch-Based Approach Using 2D Foundation Models

2026-06-17 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

CS3F is a novel training-free batch-based framework designed for zero-shot anomaly detection (ZSAD) in 3D medical images, using existing 2D foundation models. It addresses challenges like heterogeneous acquisition protocols and scarce annotated data in medical imaging. The framework decomposes 3D volumes along multiple anatomical axes, encoding slices with a 2D vision transformer. These are then converted into localized volumetric tokens via pooling, with anomaly scores derived from cross-subject mutual similarity. A coarse-to-fine tokenization strategy is introduced to mitigate focal lesion signal attenuation. CS3F was evaluated on brain MRI for metastases, glioma, and stroke, and validated on lung CT, demonstrating that frozen 2D foundation models can localize anomalies in 3D medical images. The effectiveness of fine tokenization varies with lesion contrast and imaging modality.

Key takeaway

For AI Scientists or Research Scientists developing anomaly detection in 3D medical imaging, CS3F offers a compelling training-free, zero-shot approach by effectively repurposing readily available 2D foundation models. This method directly addresses the scarcity of annotated 3D data and diverse clinical scenarios. You should carefully evaluate the coarse-to-fine tokenization strategy, as its benefit for anomaly localization is highly dependent on the specific lesion contrast and imaging modality of your application.

Key insights

CS3F enables training-free zero-shot anomaly detection in 3D medical images by adapting 2D foundation models.

Principles

• Cross-subject mutual similarity identifies anomalies.
• 2D vision transformers adapt for 3D volumes.
• Coarse-to-fine tokenization improves focal lesion detection.

Method

Decompose 3D volumes, encode slices with 2D vision transformers, pool into volumetric tokens, then score anomalies via cross-subject similarity, using a coarse-to-fine tokenization strategy.

In practice

• Apply 2D vision transformers for 3D medical image analysis.
• Use cross-subject similarity for ZSAD in heterogeneous data.
• Evaluate coarse-to-fine tokenization for subtle lesions.

Topics

• Zero-shot Anomaly Detection
• 3D Medical Imaging
• 2D Foundation Models
• Vision Transformers
• Brain MRI
• Lung CT

Best for: AI Scientist, Research Scientist

Related on AIssential

• Intelligent Healthcare Imaging Platform: A VLM-Based Framework for Automated Medical Image Analysis and Clinical Report Generation — A VLM-based framework automates medical image analysis and clinical report generation across multiple imaging modalities.
• Physics-Driven Zero-Shot MRI Reconstruction with Non-local Image Priors — Combining physics-driven constraints with non-local image priors enhances zero-shot MRI reconstruction stability and performance.
• CA-GCL: Cross-Anatomy Global-Local Contrastive Learning for Robust 3D Medical Image Understanding — CA-GCL uses global-local contrastive learning and clinical-aware text augmentation to prevent representation collapse in 3D medical FVLP.
• PEFT-MedSAM: Efficient Fine-Tuning of Medical Foundation Models for Explainable Skin Lesion Segmentation — PEFT-MedSAM efficiently fine-tunes medical foundation models for skin lesion segmentation by training only the mask decoder.
• Impact of Synthetic Lesional MR Images in Automated Focal Cortical Dysplasia Detection in Low-Data Scenarios — Synthetic FCD-MRIs generated by conditional networks can augment real data, reducing labeled data needs for automated detection.
• Vision-Language Model-Guided Deep Unrolling Enables Personalized, Fast MRI — PASS uses a VLM to guide personalized MRI sampling and reconstruction for faster, anomaly-aware imaging.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.