Consistently Informative Soft-Label Temperature for Knowledge Distillation

2023-03-30 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

CIST (Consistently Informative Soft-label Temperature) is a novel knowledge distillation framework designed to overcome the limitations of standard fixed-temperature methods. It addresses inconsistent teacher soft-label entropy and rigid teacher-student logit-scale alignment by assigning separate, sample-wise adaptive temperatures to both teacher and student models. CIST also reweights the distillation objective based on teacher confidence and student learning difficulty. Theoretically, the framework demonstrates that teacher-label entropy is primarily controlled by the ratio of the maximum teacher logit to the temperature. Empirically, CIST consistently improves performance across diverse vision tasks, including CIFAR-100 and ImageNet, and language distillation tasks, such as instruction-following benchmarks (Dolly, Vicuna, Super-NI, UnNI). It achieves significant gains, like up to +2.25% Top-1 accuracy on ImageNet, with negligible computational overhead.

Key takeaway

For Machine Learning Engineers optimizing knowledge distillation, adopting CIST can significantly enhance student model performance and stability. You should implement sample-wise adaptive temperatures for both teacher and student, normalizing dominant logits by a constant ρ, and rescale your KL distillation loss by the product of these temperatures. This approach relaxes rigid logit matching and provides a confidence-aware curriculum, leading to more consistently informative soft labels and superior results on vision and language tasks with negligible overhead.

Key insights

Fixed-temperature knowledge distillation creates inconsistent soft-label entropy, hindering effective knowledge transfer.

Principles

• Teacher-label entropy is governed by the ratio of dominant logit to temperature.
• Separate adaptive temperatures relax rigid logit-scale matching.
• Distillation loss reweighting can prioritize reliable and learnable signals.

Method

CIST assigns sample-wise adaptive temperatures to teacher and student based on dominant logits, stabilizing entropy. It then reweights the KL distillation loss by the product of these temperatures, creating a confidence-aware curriculum.

In practice

• Normalize dominant logits by a constant ρ for adaptive temperature.
• Use separate temperatures for teacher and student models.
• Rescale KL loss by τ_i^t τ_i^s for confidence-aware weighting.

Topics

• Knowledge Distillation
• Adaptive Temperature Scaling
• Soft-Label Entropy
• Logit Matching
• Vision-Language Models
• Model Compression

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Rethinking the Role of Temperature in Large Language Model Distillation — Temperature fundamentally changes FKL vs. RKL in LLM distillation, making FKL superior at higher temperatures.
• Improved Knowledge Distillation for Land-Use Image Classification — Combining hard and soft supervision with KL divergence and Cosine Similarity improves knowledge distillation for land-use classification.
• Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces — Reasoning in large output spaces can be mechanistically characterized as a two-phase process: shortlisting followed by fine-grained reasoning.
• Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models — Switch-KD unifies VLM knowledge transfer in a shared text-probability space for efficient deployment.
• Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation — DO-ALL uses Dataset Distillation to create stable, privacy-conscious source references for robust continual test-time adaptation.
• No more Catastrophic Forgetting in SFT — EKSFT selectively masks high-entropy or high-KL divergence tokens during SFT to mitigate catastrophic forgetting and preserve generalization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.