Training Agents 2: Live tutorial on model distillation for training custom agents.

2026-06-29 · Source: HuggingFace · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, quick

Summary

This live session will cover how to transfer capability from a teacher model to a smaller student through distillation. It details supervised fine-tuning on teacher-generated data (off-policy), on-policy and online methods where the teacher scores the student live, and self-distillation where the model teaches itself. The tutorial will explore distillation's four organizing axes: signal, data source, timing, and teacher identity, alongside white-box versus black-box approaches. It also frames distillation as reinforcement learning, utilizing Kullback-Leibler (KD) distance as a dense, token-level reward. All discussed methods are implemented within the TRL framework, part of the Training Agents series.

Key takeaway

For AI Engineers developing custom agents, understanding model distillation is crucial for optimizing performance and resource usage. You should explore off-policy, on-policy, and self-distillation techniques within frameworks like TRL to transfer capabilities from larger models to more efficient student agents. Consider self-distillation when aiming to improve agent performance without relying on external, stronger teachers.

Key insights

Model distillation transfers capabilities from large teacher models to smaller student models using various on-policy, off-policy, and self-distillation techniques.

Principles

• Distillation has four axes: signal, data source, timing, teacher identity.
• Self-distillation can outperform stronger teacher models.
• KD distance can serve as a token-level reward in RL.

Method

Distillation involves supervised fine-tuning on teacher-generated data (off-policy), or on-policy scoring by a teacher in a live loop, or self-distillation where the model teaches itself, all implementable in TRL.

In practice

• Use TRL for implementing distillation methods.
• Apply off-policy SFT with teacher outputs.
• Explore self-distillation for agent training.

Topics

• Model Distillation
• Agent Training
• Reinforcement Learning
• Supervised Fine-tuning
• TRL Framework
• HuggingFace

Code references

• burtenshaw/training-agents

Best for: Machine Learning Engineer, AI Engineer, AI Scientist

Related on AIssential

• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents — Guided On-Policy Distillation improves multi-turn agent capability transfer by curriculum-based teacher intervention, preventing student error compounding.
• Reinforcement Learning from Rich Feedback with Distributional DAgger — DistIL uses a forward cross-entropy objective with rich feedback to guarantee monotonic policy improvement in reinforcement learning.
• Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents — Direct self-distillation can degrade tool-use by misaligning with verifier rewards; distillation should guide credit, not replace policy gradient.
• Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning — Entropy-guided self-distillation improves LLM reasoning by adaptively weighting teacher supervision based on confidence.
• Self-Distilled Agentic Reinforcement Learning — SDAR combines gated self-distillation with reinforcement learning to stabilize and enhance LLM agent training.
• Extreme Region Policy Distillation — Decoupling RL optimization into aggressive signal extraction and constrained distillation improves both sample and KL efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by HuggingFace.