Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression

2026-03-04 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

Mixture-of-Experts (MoE) models, while efficient for scaling capacity, face significant memory bottlenecks during deployment due to their large parameter footprint. This work categorizes retraining-free MoE compression into three paradigms: Expert Pruning, Expert Editing, and Expert Merging. It identifies "router-expert mismatch" as the primary cause of persistent performance degradation after compression, arguing that current retraining-free methods neglect to update the router when experts are modified. To address this, the authors propose Router Knowledge Distillation (Router KD), a lightweight method that updates only the router parameters by distilling the original model's next-token distribution on unlabeled calibration data. Experiments on Qwen3-30B-A3B-Instruct-2507 and Mixtral-8x7B-Instruct-v0.1 demonstrate that Router KD consistently recovers performance across all three compression paradigms, with significantly larger gains observed in fine-grained MoEs like Qwen3 due to their more complex routing decision boundaries.

Key takeaway

For NLP Engineers and AI Scientists deploying compressed MoE models, integrating Router Knowledge Distillation (Router KD) is crucial to recover performance lost due to router-expert mismatch. Your teams should consider Router KD as a standard post-compression step, especially for fine-grained MoE architectures like Qwen3, as it offers substantial performance recovery with minimal computational overhead. This approach allows for efficient deployment without full model retraining.

Key insights

Router calibration is essential for effective retraining-free MoE compression, mitigating performance degradation from router-expert mismatch.

Principles

• Expert compression necessitates router calibration.
• Fine-grained MoEs benefit more from router calibration.

Method

Router Knowledge Distillation (Router KD) updates only the router parameters by distilling the original model's next-token distribution on unlabeled calibration data, minimizing computational overhead.

In practice

• Apply Router KD after MoE expert compression.
• Prioritize Router KD for fine-grained MoE architectures.

Topics

• Mixture-of-Experts
• MoE Compression
• Router Calibration
• Knowledge Distillation
• Large Language Models

Best for: NLP Engineer, AI Scientist, Research Scientist, AI Researcher, AI Engineer, Machine Learning Engineer

Related on AIssential

• Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models — MoE quantization requires preserving both router output values and structural relationships to maintain expert selection consistency and mitigate performance degradation.
• Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails — Knowledge Distillation (KD) excels in low-data LLM post-training, especially with strong teachers, and can be enhanced via a two-stage synthetic data strategy.
• When Model Merging Breaks Routing: Training-Free Calibration for MoE — MoE model merging often fails due to routing breakdown; HARC offers a training-free, curvature-aware calibration to fix it.
• Sub-Token Routing in LoRA for Adaptation and Query-Aware KV Compression — Sub-token routing offers fine-grained control for transformer efficiency by compressing within token representations.
• MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers — MoECa accelerates DiT-MoE inference by reusing expert-branch features across timesteps, achieving up to 2.83x speedup.
• KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models — KBVQ-MoE significantly improves ultra-low-bit quantization for MoE LLMs by eliminating expert redundancy and correcting output bias.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.