Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Computer Vision · Depth: Expert, extended

Summary

Few-medoids is an embarrassingly simple coreset selection strategy designed for few-shot knowledge distillation (KD) that addresses computational inefficiencies in deep learning. This method identifies a small, highly representative subset of a dataset by selecting samples closest to the centroid of each class within the teacher model's latent space. Researchers conducted extensive KD experiments across four datasets—CIFAR-10, CIFAR-100, Oxford Flowers 102, and Food-101—and three teacher-student model pairs, including ResNet-34→ResNet-18, ViT-B/16→ViT-Small, and ViT-B/16→ResNet-50. Empirical results consistently show few-medoids outperforming random selection, herding, and k-center Greedy baselines, especially in low-to-mid coreset dimensions (k ≤ 32) and when the student model is trained from scratch. The method's code is publicly released at https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset.

Key takeaway

For Machine Learning Engineers optimizing deep learning training efficiency, few-medoids offers a robust, simple coreset selection method for few-shot knowledge distillation. If you are training student models from scratch, you should consider implementing few-medoids to consistently outperform traditional baselines like random selection or herding, reducing computational burden while maintaining or improving student performance. Explore its application, particularly in low-to-mid coreset dimensions, to streamline your model development.

Key insights

Few-medoids, a simple geometric centrality method in the teacher's latent space, consistently outperforms complex coreset selection for few-shot knowledge distillation.

Principles

• Coreset selection reduces training computational overhead.
• Geometric centrality identifies representative data samples.
• Semantically relevant teacher supervision improves KD.

Method

Few-medoids calculates each sample's average Euclidean distance to other class members in the teacher's latent space. It ranks samples by these scores, selecting the top k most central ones per class to form the coreset for student training.

In practice

• Use few-medoids as a drop-in coreset baseline.
• Apply few-medoids for few-shot KD with scratch-trained students.
• Leverage teacher's latent space for sample representativeness.

Topics

• Few-Medoids
• Coreset Selection
• Knowledge Distillation
• Few-Shot Learning
• Latent Space Analysis
• Image Classification

Code references

• CemilAndreiDilmac/Few-Shot-KD-Coreset

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation — Few-Medoids, a simple centroid-based coreset selection, consistently outperforms random and other strategies in few-shot knowledge distillation.
• Rethinking the Good Enough Embedding for Easy Few-Shot Learning — High-quality, off-the-shelf embeddings are sufficient for state-of-the-art few-shot classification, bypassing complex meta-learning.
• TAROT: Task-Adaptive Refinement of LLM-prior Graphs for Few-shot Tabular Learning — TAROT refines LLM-generated semantic graphs for few-shot tabular learning, enhancing predictive performance by mitigating hallucination and capturing feature relationships.
• Efficient Neural Network Model Selection for Few-Class Application Datasets — "Few-class distinctiveness" is a data-side metric enabling faster, more efficient neural network model selection for datasets with few classes.
• MixSD: Mixed Contextual Self-Distillation for Knowledge Injection — Aligning supervision targets with an LLM's native generation distribution significantly mitigates catastrophic forgetting during knowledge injection.
• Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces — Reasoning in large output spaces can be mechanistically characterized as a two-phase process: shortlisting followed by fine-grained reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.