Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

2026-07-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

EcoSpec, a novel cost-aware speculative decoding framework, addresses inference inefficiencies in large-scale Sparse Mixture-of-Experts (MoE) Large Language Models. Traditional speculative decoding (SD) often causes "expert scattering" by activating disjoint experts for high-probability draft tokens, which increases expert-weight memory traffic and reduces speedup. EcoSpec mitigates this by integrating predicted marginal expert activation cost into its draft selection process. Utilizing a lightweight expert predictor and a dynamic expert buffer, EcoSpec prioritizes draft paths that reuse experts already active in the current verification set, while preserving high acceptance likelihood. Evaluated on models like DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B across diverse benchmarks, EcoSpec consistently reduced active expert footprints and achieved up to a 1.62x decoding speedup. This demonstrates the importance of considering expert activation cost for efficient MoE LLM inference.

Key takeaway

For Machine Learning Engineers optimizing large-scale Mixture-of-Experts (MoE) LLM inference, you should prioritize cost-aware speculative decoding. Implementing frameworks like EcoSpec, which consider expert activation costs during draft selection, can significantly reduce expert-weight memory traffic. This approach yields up to a 1.62x speedup on models such as DeepSeek-V3.1 (671B), directly improving deployment efficiency. Evaluate your current speculative decoding strategy for potential "expert scattering" and integrate expert reuse mechanisms to enhance performance.

Key insights

Cost-aware speculative decoding, like EcoSpec, optimizes MoE LLM inference by minimizing expert activation costs during draft selection.

Principles

• MoE inference efficiency depends on expert activation patterns.
• "Expert scattering" in SD increases memory traffic.
• Cost-aware draft selection improves MoE decoding speed.

Method

EcoSpec integrates predicted marginal expert activation cost into draft selection. It uses a lightweight expert predictor and dynamic expert buffer to favor draft paths reusing existing experts, without modifying target-model verification.

In practice

• Implement cost-aware draft selection for MoE LLMs.
• Utilize expert predictors to guide SD expert reuse.
• Reduce expert-weight memory traffic in MoE inference.

Topics

• Mixture-of-Experts
• Speculative Decoding
• LLM Inference
• Expert Activation Cost
• DeepSeek-V3.1
• Qwen3-235B-A22B

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts — EcoSpec balances draft acceptance likelihood with marginal expert activation cost for MoE inference.
• Why MoE models get more from speculative decoding - Cohere — MoE sparsity and expert routing significantly enhance speculative decoding performance, especially at moderate batch sizes.
• assigning Moe to Gpus to reduce inference and memory usage — Optimizing MoE expert-to-GPU assignment using training logs is a critical challenge for LLM inference efficiency.
• A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference — Expert requests in MoE LLMs exhibit strong spatio-temporal correlations, enabling predictable prefetching for efficiency.
• Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing — Window-level optimization via reinforcement learning significantly improves speculative decoding efficiency for LLMs.
• Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study — On bandwidth-bound edge hardware, MoE inference cost tracks total parameters, not active ones, negating expected efficiency gains.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.