EMO: Frustratingly Easy Progressive Training of Extendable MoE

2026-05-13 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

EMO, a novel progressive training framework, addresses the practical inefficiencies of Sparse Mixture-of-Experts (MoE) models, which often suffer from high memory and communication costs despite their theoretical FLOPs advantage. The framework posits that early-stage training over-allocates experts, leading to bottlenecks. EMO treats MoE capacity as expandable memory, progressively growing the expert pool throughout the training process. It models sparsity within scaling laws to determine compute-optimal token budgets for each expansion stage. Empirical evaluations demonstrate that EMO achieves performance comparable to fixed-expert MoE configurations in large-scale experiments, while simultaneously enhancing wall-clock efficiency, reducing both training time and GPU expenditure.

Key takeaway

For Research Scientists developing large-scale MoE models, you should consider adopting progressive training frameworks like EMO. This approach can significantly reduce your GPU costs and training times by dynamically scaling expert capacity, allowing you to achieve comparable performance to fixed-expert setups more efficiently.

Key insights

Progressively expanding MoE expert pools during training improves efficiency without sacrificing performance.

Principles

• MoE capacity can be treated as expandable memory.
• Early-stage training often over-allocates experts.

Method

EMO progressively expands the expert pool, modeling sparsity in scaling laws to derive stage-wise compute-optimal token budgets for expansion.

In practice

• Reduce MoE training time.
• Lower GPU costs for MoE models.

Topics

• EMO Framework
• Mixture-of-Experts
• Progressive Training
• Scaling Laws
• Training Efficiency

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts — Expert upcycling efficiently scales MoE models by duplicating experts mid-training, preserving inference cost and leveraging warm initialization.
• Post-Trained MoE Can Skip Half Experts via Self-Distillation — ZEDA efficiently converts static MoE models to dynamic ones, reducing inference costs via self-distillation.
• A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference — Expert requests in MoE LLMs exhibit strong spatio-temporal correlations, enabling predictable prefetching for efficiency.
• Boosting MoE Training Throughput with Advanced Fusion Kernels — Fusing MoE block operations with custom kernels significantly boosts training throughput by eliminating bottlenecks.
• Mixture of Experts (MoE) - More Parameters, Same Compute — Mixture of Experts scales model capacity by having many specialized networks, but keeps per-token compute low by activating only a few.
• Streaming experts — Streaming experts allows large MoE models to run on RAM-limited hardware by loading weights from SSD on demand.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.