Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

2026-07-02 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, medium

Summary

Generic TB-Coverage is a novel method for pruning SparseMixture-of-Experts (MoE) language models, addressing the challenge of structured redundancy without requiring downstream calibration data. Unlike existing techniques that rely on a single aggregated importance score, this approach profiles per-expert utility separately across generic text corpora like WikiText2 and C4. It then applies a fixed-budget coverage rule to preserve high-utility experts from each corpus, constructing a final pruning mask. Evaluated on Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at 25%, 50%, and 75% retention budgets, Generic TB-Coverage significantly improves average accuracy on six common zero-shot benchmarks compared to random pruning, REAP, and ExpertSparsity. It also reduces perplexity degradation on WikiText2 and C4, with the most substantial gains observed under aggressive pruning at 25% and 50% retention.

Key takeaway

For Machine Learning Engineers optimizing Sparse Mixture-of-Experts (MoE) models, especially with limited downstream calibration data, you should consider coverage-aware pruning. Generic TB-Coverage, using generic corpora, enables effective expert reduction. This approach improves accuracy and reduces perplexity degradation, particularly at 25% to 50% retention. Implement this to achieve significant memory and inference cost reductions while maintaining model performance.

Key insights

Preserving cross-corpus expert coverage with generic data improves MoE pruning without downstream calibration.

Principles

• Expert utility should be profiled separately per corpus.
• Cross-corpus expert coverage is a strong generic-data prior.

Method

Profile per-expert utility on generic text corpora (e.g., WikiText2, C4), then apply a fixed-budget coverage rule to preserve high-utility experts from each corpus for the pruning mask.

In practice

• Utilize WikiText2 and C4 for generic calibration.
• Prioritize coverage-aware pruning for aggressive MoE compression.

Topics

• Sparse Mixture-of-Experts
• Expert Pruning
• Language Models
• Model Compression
• Zero-shot Benchmarks
• Generic Data Calibration

Code references

• imagination-research/EEP
• ZongfangLiu/EvoESAP
• jerryy33/MoE_analysis

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts — DAG-MoE improves Mixture-of-Experts performance by replacing weighted-summation with learned structural aggregation, expanding expert-combination space and enabling multi-step reasoning.
• SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling — SPRI leverages SVD-partitioned residuals from pretrained FFNs to create diverse MoE experts, improving upcycling under data constraints.
• Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts — FC-MoE resolves expert conflicts in federated MoE LLM fine-tuning, improving stability and performance in non-IID settings.
• How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle — A unified formulation provides a principle for selecting optimal MoE expert pruning criteria based on task requirements.
• Less is MoE: Trimming Experts in Domain-Specialist Language Models — Fisher importance-based pruning of FFN intermediate dimensions effectively compresses MoE models, preserving capability and improving inference.
• Inside the Sparse Brain: How Mixture-of-Experts (MoE) Makes LLMs Smarter, Faster, and Greener — Mixture-of-Experts enables efficient, large-scale AI by activating only specialized model components per task.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.