How Small Models Learn to Think Like Giants

2026-07-15 · Source: Jia-Bin Huang · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, long

Summary

Knowledge distillation transfers capabilities from large "teacher" models to smaller "student" models, making them faster and more affordable. Initially, this involved token-level distillation using "dark knowledge" from teacher probability distributions, softened by temperature, and minimizing KL divergence. However, limitations like exposure bias in sequence-level distillation, where students train on teacher trajectories but infer on their own, led to compounding errors. Reinforcement learning addresses this by training on student trajectories but suffers from sparse reward signals. On-policy distillation combines the strengths, allowing student models to generate their own trajectories while receiving dense, token-level feedback from a teacher. This approach, demonstrated with models like Qwen 38B and T5 XL, significantly improves performance, compute efficiency (9-30x less estimated compute), and addresses challenges like catastrophic forgetting in personalization and effective data reuse. On-Policy Self-Distillation (OPSD) further refines this by enabling a single model to act as both teacher and student using privileged context.

Key takeaway

For Machine Learning Engineers developing efficient or specialized LLMs, on-policy distillation offers a powerful solution to improve smaller models. You should consider implementing this approach to train on your model's own generated trajectories, leveraging dense, token-level feedback from a teacher or even through self-distillation. This method significantly boosts performance, reduces compute costs by 9 to 30 times compared to supervised fine-tuning, and effectively addresses challenges like catastrophic forgetting and data reuse in personalization tasks.

Key insights

Knowledge distillation effectively transfers capabilities from large teacher models to smaller student models, enhancing efficiency and performance.

Principles

• Softened teacher probability distributions (dark knowledge) provide richer training signals.
• On-policy training mitigates exposure bias by learning from student-generated states.
• Dense, token-level feedback on student trajectories is more efficient than sparse rewards.

Method

On-policy distillation involves a student model generating its own trajectories, with a powerful teacher assigning dense, token-level feedback on the student's actual outputs. This allows training on partial rollouts and uses a single teacher forward pass.

In practice

• Apply on-policy distillation to teach mathematical reasoning, achieving 70% accuracy on benchmarks.
• Restore instruction following (IFE eval to 83%) in personalized models after fine-tuning on private data.
• Enable effective data reuse by generating varied student responses for dense teacher feedback.

Topics

• Knowledge Distillation
• On-Policy Distillation
• Self-Distillation
• Exposure Bias
• LLM Personalization
• Continual Learning

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• AI 101: "On-Policy Distillation Zeitgeist" — Self-distillation offers a scalable, on-policy method for LLMs to refine reasoning by self-critique and dense feedback.
• A Survey of On-Policy Distillation for Large Language Models — OPD overcomes off-policy distillation's exposure bias by enabling LLMs to learn from self-generated trajectories with teacher feedback.
• The Sequence Knowledge #886: Demystifying Model Distillation — Knowledge distillation trains a small "student" model by having it learn from a large "teacher" model's interpretations, alongside the original data.
• Self-Policy Distillation via Capability-Selective Subspace Projection — Self-Policy Distillation (SPD) enhances LLMs by selectively distilling task-relevant capabilities from self-generated data without external signals.
• Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding — Quality-aware self-distillation enhances GUI grounding by filtering unreliable teacher signals and calibrating supervision strength.
• Self distillation — Self-distillation improves models by creating superior training targets through systemic asymmetry, not by generating new information.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Jia-Bin Huang.