Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A new expert pruning method, Generic TB-Coverage, addresses structured redundancy in Sparse Mixture-of-Experts (MoE) language models. Existing methods often bias retained experts by using single aggregated importance scores. Generic TB-Coverage overcomes this by profiling per-expert utility separately on generic text corpora like WikiText2 and C4, then applying a fixed-budget coverage rule to preserve high-utility experts from each. Tested on Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at 25%, 50%, and 75% retention, the method significantly improves average accuracy on six zero-shot benchmarks compared to random pruning, REAP, and ExpertSparsity. It also reduces perplexity degradation on WikiText2 and C4, with the largest gains observed under aggressive pruning (25% and 50% retain) without downstream calibration data.

Key takeaway

For Machine Learning Engineers optimizing Sparse Mixture-of-Experts models, especially when downstream calibration data is unavailable, you should consider implementing the Generic TB-Coverage method. This approach, which leverages generic text corpora and a coverage-aware pruning rule, demonstrably improves accuracy and reduces perplexity degradation, particularly under aggressive pruning budgets (25% and 50% retention). Adopting this method can lead to more robust and efficient MoE deployments.

Key insights

Preserving cross-corpus expert coverage with generic data effectively prunes Sparse Mixture-of-Experts language models.

Principles

• MoE LMs contain substantial structured redundancy.
• Single aggregated importance scores bias expert pruning.
• Cross-corpus expert coverage improves MoE pruning efficacy.

Method

Generic TB-Coverage profiles per-expert utility on generic corpora (WikiText2, C4) and enforces a fixed-budget coverage rule to select high-utility experts from each corpus before constructing the final pruning mask.

In practice

• Calibrate MoE pruning with WikiText2 and C4.
• Profile expert utility separately per corpus.
• Apply coverage-aware rules for aggressive pruning.

Topics

• Sparse Mixture-of-Experts
• Language Models
• Model Pruning
• Expert Coverage
• Qwen1.5-MoE-A2.7B
• DeepSeek-MoE-16B-Base
• Zero-shot Benchmarks

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle — A unified formulation provides a principle for selecting optimal MoE expert pruning criteria based on task requirements.
• ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference — ProxyKV uses an asynchronous small-model proxy to prune KV caches, achieving high accuracy and significant speedups for long-context LLM inference.
• Intro to Mixture of Experts | Aritra Roy Gosthipaty | HF Podcast #2 — Mixture of Experts (MOEs) enhance LLM efficiency by sparsely activating parameters, making them ideal for large foundational models.
• Less is MoE: Trimming Experts in Domain-Specialist Language Models — Fisher importance-based pruning of FFN intermediate dimensions effectively compresses MoE models, preserving capability and improving inference.
• MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers — MoECa accelerates DiT-MoE inference by reusing expert-branch features across timesteps, achieving up to 2.83x speedup.
• Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation — MoE language specialization emerges in final layers, enabling efficient adaptation by targeting those layers.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.