Path-Constrained Mixture-of-Experts

2026-07-06 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Path-Constrained Mixture-of-Experts (PathMoE) is a new architecture that views Sparse Mixture-of-Experts (MoE) computation through expert paths, which are sequences of expert selections across layers. This approach observes that tokens naturally cluster into a small fraction of N^L possible paths, indicating statistical inefficiency. PathMoE addresses this by constraining the effective path space, specifically by sharing router parameters across blocks of consecutive layers. This design amplifies the emergent path structure, leading to more concentrated path clusters, improved cross-layer consistency, and enhanced robustness to routing perturbations. Experiments with 0.9B and 16B parameter PathMoE models demonstrate consistent improvements in perplexity and downstream tasks compared to independent routing, while also eliminating the need for auxiliary losses. This establishes expert paths as a valuable design axis for MoE architectures.

Key takeaway

For Machine Learning Engineers designing or optimizing Mixture-of-Experts (MoE) models, consider adopting PathMoE architectures. This approach constrains expert paths by sharing router parameters across layers. It consistently improves perplexity and downstream task performance for models up to 16B parameters. You can also eliminate the need for auxiliary routing losses, simplifying training and enhancing robustness to routing perturbations. Evaluate PathMoE to achieve more efficient and stable sparse MoE deployments.

Key insights

Path-Constrained Mixture-of-Experts (PathMoE) improves MoE efficiency and performance by explicitly managing and amplifying token routing paths across layers.

Principles

• Tokens naturally cluster into few expert paths.
• Constraining path space amplifies routing concentration.
• Expert paths offer a new MoE design axis.

Method

PathMoE constrains expert path space by sharing router parameters across blocks of consecutive layers, amplifying emergent path structure and improving cross-layer consistency.

In practice

• Improve perplexity in 0.9B and 16B MoE models.
• Eliminate auxiliary losses in MoE training.
• Enhance robustness to routing perturbations.

Topics

• PathMoE
• Mixture-of-Experts
• Expert Routing
• Sparse Models
• Language Model Performance
• Model Efficiency

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• MoE Token Routing Explained: How Mixture of Experts Works (with Code) — Token routing in MoE architectures selectively directs tokens to a subset of experts, ensuring sparsity and computational efficiency.
• Eradicating Negative Transfer in Multi-Physics Foundation Models via Sparse Mixture-of-Experts Routing — Sparse Mixture-of-Experts routing effectively mitigates negative transfer in multi-physics foundation models.
• DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts — DAG-MoE improves Mixture-of-Experts performance by replacing weighted-summation with learned structural aggregation, expanding expert-combination space and enabling multi-step reasoning.
• Redesign Mixture-of-Experts Routers with Manifold Power Iteration — Manifold Power Iteration (MPI) aligns MoE router rows with expert principal singular directions for improved token-expert affinity.
• RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts — Optimizing MoE inference requires dynamic kernel configuration based on runtime expert routing, not just batch size.
• STAR: Rethinking MoE Routing as Structure-Aware Subspace Learning — STAR rethinks MoE routing as structure-aware subspace learning for stable expert specialization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.