Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks

2026-05-18 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

A study benchmarked nine self-supervised learning (SSL) methods across four pretext-task families, pretraining them on 10,412 3D CT scans (1.89 million 2D axial slices). Each pretrained Swin Transformer encoder was integrated into a SwinUNETR-style segmentation network and fine-tuned on nine public segmentation tasks, including large abdominal organs, head-and-neck structures, and tumors from CT and MRI. Performance was evaluated using the Dice similarity coefficient (DSC), convergence speed, cross-modality transferability (CT-to-MRI), and feature-reuse patterns. Self-distilled masked image transformer (SMIT) achieved the highest overall segmentation accuracy, fastest fine-tuning convergence, and smallest few-shot-to-many-shot performance gap, demonstrating superior data efficiency and consistent feature reuse.

Key takeaway

For research scientists developing medical image segmentation models, especially with limited labeled data, you should prioritize self-distilled masked image transformer (SMIT) pretraining. SMIT demonstrates superior data efficiency and faster convergence, making it a strong candidate for achieving high accuracy and consistent feature reuse across various anatomical structures and modalities like CT and MRI.

Key insights

SMIT, combining MIM and self-distillation, excels in medical image segmentation transfer learning.

Principles

• MIM and self-distillation outperform contrastive learning.
• SSL choice matters most with limited annotation budgets.

Method

Pretrain Swin Transformer encoders with SSL on 3D CT scans, then fine-tune within a SwinUNETR-style network for segmentation tasks.

In practice

• Prioritize SMIT for medical image segmentation.
• Focus on SSL method selection for few-shot scenarios.

Topics

• Self-Supervised Learning
• Image Segmentation
• Swin Transformer
• Masked Image Modeling
• SMIT

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Masked Autoencoder for Image Reconstruction using Vision Transformers (PyTorch) — Masked Autoencoders with Vision Transformers learn robust image representations by reconstructing heavily masked image patches.
• SAM3 Self-Distillation for Fine-Grained GOOSE 2D Semantic Segmentation — Self-distillation using SAM3 as a teacher, combined with image-level multi-scale augmentation and aggressive photometric distortion, significantly boosts fine-grained semantic segmentation.
• Efficient Transformer-Based Localized Patch Sampling for Choroid Plexus Segmentation in Multiple Sclerosis — A SwinUNETR pipeline with localized patch sampling accurately segments choroid plexus in MS with 99% less computational cost.
• Single-Stage Hierarchical Rectification for Weakly Supervised Histopathology Segmentation — SSHR proactively purifies features in a single stage, outperforming multi-stage WSSS and reducing training time significantly.
• Efficient Remote Sensing Instance Segmentation with Linear-Time State Space Distilled Visual Foundation Models — Distilling knowledge into linear-time state space models significantly boosts remote sensing instance segmentation efficiency.
• A 3D SAM-Based Progressive Prompting Framework for Multi-Task Segmentation of Radiotherapy-induced Normal Tissue Injuries in Limited-Data Settings — A progressive prompting framework enhances 3D SAM for accurate, multi-task segmentation of small, sparse radiotherapy injuries with limited data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.