KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models

2026-07-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, long

Summary

KDFlow is a novel framework designed for efficient knowledge distillation (KD) of large language models (LLMs). It addresses the suboptimal training efficiency of existing frameworks by employing a decoupled architecture, assigning the teacher model to the high-throughput inference engine SGLang and the student model to the dedicated training engine PyTorch FSDP2. KDFlow achieves this by transferring the teacher's compact hidden states using zero-copy data transfer and recomputing full logits on the student side, balancing communication cost and KD performance. This approach yields a 1.44$\times$ to 6.36$\times$ speedup compared to current KD frameworks, especially for Mixture-of-Experts (MoE) teachers, without compromising downstream task performance. The framework also supports off-policy, on-policy, and cross-tokenizer distillation through user-friendly APIs.

Key takeaway

For AI Scientists and Machine Learning Engineers focused on LLM compression, KDFlow offers a critical solution to efficiency bottlenecks. By decoupling teacher inference and student training, KDFlow delivers significant speedups (1.44$\times$ to 6.36$\times$) without sacrificing distillation quality, particularly for MoE teachers. You should consider integrating KDFlow into your research or development pipeline to accelerate experimentation and deployment of smaller, performant LLMs.

Key insights

KDFlow decouples LLM KD, using optimized backends for teacher inference (SGLang) and student training (FSDP2) to boost efficiency.

Principles

• Decouple inference and training backends for efficiency.
• Transfer hidden states, not full logits, to reduce communication.
• Recompute logits locally to preserve mathematical equivalence.

Method

KDFlow uses Ray to coordinate a Trainer, RolloutActorGroup (SGLang), TeacherActorGroup (SGLang), and StudentActorGroup (FSDP2) for decoupled KD workflows.

In practice

• Accelerate LLM distillation by 1.44$\times$-6.36$\times$.
• Support off-policy, on-policy, and cross-tokenizer KD.
• Prototype KD algorithms with minimal engineering overhead.

Topics

• Knowledge Distillation
• Large Language Models
• Model Compression
• Distributed Training
• SGLang
• PyTorch FSDP2
• Mixture-of-Experts

Code references

• songmzhang/KDFlow
• huggingface/trl
• THUDM/slime

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• KD4MT: A Survey of Knowledge Distillation for Machine Translation — KD in Machine Translation is a versatile knowledge transfer mechanism, not solely a compression technique.
• StreamKL: Fast and Memory-Efficient KL Divergence for Boosting Attention Distillation — StreamKL enables memory-efficient, long-context attention distillation by eliminating quadratic materialization of attention distributions through an online KL reduction.
• KDW at SemEval-2026 Task 12: Logic-Driven Distillation with Knowledge Graphs for Efficient Abductive Reasoning — KDW uses KG-driven knowledge distillation to enable efficient abductive reasoning in compact models, matching LLM performance at lower cost.
• Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations — Student-guided perturbations and cyclic shifts enable efficient, diverse knowledge distillation in a single training stage.
• A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions — Knowledge Distillation in LLMs works by sparsifying interactions, especially complex ones, retaining salient teacher knowledge.
• Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation — Pedagogically-inspired knowledge distillation systematically improves student model performance and efficiency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.