SAM3 Self-Distillation for Fine-Grained GOOSE 2D Semantic Segmentation

2026-06-18 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

A 4th-place entry in the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge achieved a composite mean Intersection-over-Union (mIoU) of 69.73% on the 1,815-image test set. The model adapts the Segment Anything Model 3 (SAM3) image encoder with a lightweight decoder. Key contributions include a novel self-distillation scheme that re-uses SAM3, prompted with ground-truth boxes, as a teacher for classes where it outperforms the custom model. Additionally, an image-level multi-scale test-time augmentation method was introduced, which restores multi-scale inference for fixed-input-size models by rescaling the image itself. An aggressive photometric distortion technique, transplanted from a winning 2025 GOOSE 2D entry, was identified as the single largest source of performance improvement.

Key takeaway

For Machine Learning Engineers optimizing fine-grained semantic segmentation, consider integrating a self-distillation strategy where powerful foundation models like SAM3 act as teachers. You should also explore image-level multi-scale test-time augmentation to restore multi-scale inference without complex model changes. Furthermore, aggressively applying photometric distortions, even those from prior challenge winners, can provide substantial performance uplifts for your pipeline.

Key insights

Self-distillation using SAM3 as a teacher, combined with image-level multi-scale augmentation and aggressive photometric distortion, significantly boosts fine-grained semantic segmentation.

Principles

• Re-use powerful foundation models as teachers.
• Image-level scaling enables multi-scale inference.
• Aggressive photometric distortion can yield large gains.

Method

The approach involves adapting a SAM3 image encoder with a lightweight decoder, applying a self-distillation scheme using SAM3 as a teacher, and employing image-level multi-scale test-time augmentation.

In practice

• Apply SAM3 as a self-distillation teacher.
• Implement image-level multi-scale TTA.
• Experiment with aggressive photometric distortions.

Topics

• Semantic Segmentation
• SAM3
• Self-Distillation
• Test-Time Augmentation
• Photometric Distortion
• Computer Vision Challenges

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Technical Report for ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Leveraging DINOv3 for Robust Outdoor Scene Understanding in Field Robotics — Combining DINOv3, Mask2Former, and test-time augmentation significantly boosts fine-grained semantic segmentation performance in challenging outdoor environments.
• Learning with Semantic Priors: Stabilizing Point-Supervised Infrared Small Target Detection via Hierarchical Knowledge Distillation — VFM-driven hierarchical knowledge distillation stabilizes point-supervised ISTD by injecting robust semantics into lightweight CNNs.
• Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks — SMIT, combining MIM and self-distillation, excels in medical image segmentation transfer learning.
• Feasibility of Indoor Frame-Wise Lidar Semantic Segmentation via Distillation from Visual Foundation Model — Cross-modal distillation from Visual Foundation Models enables effective, label-free indoor lidar semantic segmentation.
• We’re releasing SAM 3.1: a drop-in update to SAM 3 that introduces object multiplexing to significantly improve video processing efficiency without sacrificing accuracy. We’re sharing this update with the community to help make high-performance applications f - x.com — SAM 3.1 improves video processing efficiency via object multiplexing, enabling high-performance AI on smaller hardware.
• Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation — Semantic segmentation and an efficient distillation framework enable practical, scalable block attention for long-context LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.