Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Few-Medoids, an embarrassingly simple coreset selection strategy, has been introduced to efficiently identify highly representative data subsets for model training, particularly within few-shot knowledge distillation (KD) setups. This method selects samples closest to the centroid (average image) of each class, addressing the common challenge where typical sample selection strategies often fail to outperform random baselines. Extensive KD experiments were conducted across four diverse image classification datasets and three teacher-student model pairs, encompassing both convolutional and transformer networks. The empirical results consistently demonstrate that Few-Medoids surpasses both random selection and other established coreset selection strategies. Consequently, Few-Medoids is proposed as a direct drop-in replacement for existing baselines such as herding or k-center Greedy in future coreset selection research. The code is publicly available at https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset.

Key takeaway

For Machine Learning Engineers optimizing few-shot knowledge distillation, Few-Medoids provides a demonstrably effective and simple coreset selection method. If you are struggling to surpass random baselines with existing strategies like herding or k-center Greedy, you should integrate this centroid-based approach. Its consistent empirical superiority across diverse datasets and model architectures suggests it can significantly enhance training efficiency and student model performance. Consider adopting Few-Medoids as a direct replacement in your current workflows.

Key insights

Few-Medoids, a simple centroid-based coreset selection, consistently outperforms random and other strategies in few-shot knowledge distillation.

Principles

• Simplicity can yield superior performance.
• Centroid proximity indicates representativeness.
• Effective coreset selection boosts KD efficiency.

Method

Few-Medoids selects samples closest to the centroid (average image) of each class to form a representative coreset for few-shot knowledge distillation.

In practice

• Implement Few-Medoids for few-shot KD.
• Replace Herding or k-center Greedy.
• Use for efficient image classification training.

Topics

• Few-Shot Learning
• Knowledge Distillation
• Coreset Selection
• Image Classification
• Machine Learning Efficiency
• Deep Learning

Code references

• CemilAndreiDilmac/Few-Shot-KD-Coreset

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning — CLIMB combines hierarchical centroid-based memory with knowledge distillation for robust online continual self-supervised learning.
• Efficient Neural Network Model Selection for Few-Class Application Datasets — "Few-class distinctiveness" is a data-side metric enabling faster, more efficient neural network model selection for datasets with few classes.
• Rethinking the Good Enough Embedding for Easy Few-Shot Learning — High-quality, off-the-shelf embeddings are sufficient for state-of-the-art few-shot classification, bypassing complex meta-learning.
• TAROT: Task-Adaptive Refinement of LLM-prior Graphs for Few-shot Tabular Learning — TAROT refines LLM-generated semantic graphs for few-shot tabular learning, enhancing predictive performance by mitigating hallucination and capturing feature relationships.
• Automatic Combination of Sample Selection Strategies for Few-Shot Learning — Combining diverse sample selection strategies significantly boosts few-shot learning, especially at low shot counts.
• Entropy Centroids as Intrinsic Rewards for Test-Time Scaling — Temporal patterns of LLM uncertainty, captured by Entropy Centroids, predict response quality without external rewards.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.