PEFT-MedSAM: Efficient Fine-Tuning of Medical Foundation Models for Explainable Skin Lesion Segmentation

2026-06-17 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition, Health & Medical Research · Depth: Advanced, quick

Summary

PEFT-MedSAM introduces a parameter-efficient fine-tuning method designed to adapt the Medical Segment Anything Model (MedSAM) for automated segmentation of dermoscopic skin lesions. This approach freezes MedSAM's pre-trained image and prompt encoders, focusing training solely on the lightweight mask decoder. Experiments on the ISIC 2018 benchmark dataset demonstrated a Dice coefficient of .9411 and an Intersection over Union (IoU) of .8918, outperforming a fully trained U-Net baseline (.8715 Dice) and zero-shot MedSAM inference (.8997 Dice). External validation using the PH2 dataset yielded a .9467 Dice coefficient with a standard deviation of +/- .0310. Statistical analysis confirmed significance with a p-value less than .0001. Furthermore, Grad-CAM explainability and a pointing game evaluation showed 98.27% accuracy on a 519-image validation set, confirming accurate lesion region classification to enhance clinical trustworthiness.

Key takeaway

For AI Scientists and Machine Learning Engineers developing medical image segmentation models, PEFT-MedSAM offers a highly efficient fine-tuning strategy. You should consider this method to adapt large foundation models like MedSAM for specific tasks, such as skin lesion detection, significantly reducing computational overhead while achieving superior performance. This approach also integrates explainability, crucial for building clinical trust and accelerating deployment in diagnostic workflows.

Key insights

PEFT-MedSAM efficiently fine-tunes medical foundation models for skin lesion segmentation by training only the mask decoder.

Principles

• Parameter-efficient tuning excels in medical imaging.
• Freezing encoders preserves pre-trained knowledge.
• Explainability boosts clinical trust.

Method

PEFT-MedSAM fine-tunes MedSAM by freezing its pre-trained image and prompt encoders, training only the lightweight mask decoder. It uses Grad-CAM for explainability and a pointing game for evaluation.

In practice

• Apply PEFT to adapt large medical vision models.
• Use Grad-CAM for model interpretability in clinical AI.
• Benchmark against zero-shot and fully trained baselines.

Topics

• PEFT-MedSAM
• Skin Lesion Segmentation
• Medical Foundation Models
• Parameter-Efficient Fine-Tuning
• Grad-CAM Explainability
• Dermoscopic Images

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Mutual Distillation of Dual-Foundation Models for Semi-Supervised PET/CT Segmentation — Mutual distillation of dual-foundation models improves semi-supervised PET/CT segmentation with minimal labeled data.
• EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models — EasyLens enhances medical VLM sensitivity to subtle lesions by amplifying relevant patch representations without retraining.
• SegGuidedNet: Sub-Region-Aware Attention Supervision for Interpretable Brain Tumor Segmentation — SegGuidedNet achieves accurate brain tumor sub-region segmentation with built-in interpretability through sub-region-aware attention supervision.
• Med-CAM: Minimal Evidence for Explaining Medical Decision Making — Med-CAM generates minimal, precise, and faithful evidence maps for medical AI decisions by matching classifier activations.
• Toward Training-Free Zero-Shot Anomaly Detection in 3D Medical Images: A Batch-Based Approach Using 2D Foundation Models — CS3F enables training-free zero-shot anomaly detection in 3D medical images by adapting 2D foundation models.
• Evidential learning driven Breast Tumor Segmentation with Stage-divided Vision-Language Interaction — TextBCS enhances breast tumor segmentation in MRI by combining text prompts with evidential learning to address low contrast and blurred boundaries.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.