Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

2026-07-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

EcoSpec is a novel cost-aware speculative decoding framework designed to enhance the inference efficiency of large-scale Mixture-of-Experts (MoE) models. It addresses the "expert scattering" problem, where confidence-driven speculative decoding in MoE models can activate disjoint sets of experts, increasing memory traffic and reducing speedup. EcoSpec integrates a lightweight expert predictor and a dynamic expert buffer into the draft selection process, prioritizing candidate token paths that reuse existing experts while preserving high acceptance likelihood. This approach avoids modifying the target-model verification rule. Evaluated on DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B across diverse benchmarks, EcoSpec consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to 1.62x speedup.

Key takeaway

For machine learning engineers deploying large Mixture-of-Experts (MoE) language models, you should re-evaluate speculative decoding strategies. Standard methods, focused solely on acceptance likelihood, can lead to "expert scattering" and increased memory traffic, hindering actual speedup. Implement cost-aware draft selection, such as EcoSpec's approach, to explicitly balance acceptance probability with marginal expert activation cost. This will reduce verification latency and HBM traffic, significantly improving end-to-end inference throughput for your MoE models.

Key insights

EcoSpec balances draft acceptance likelihood with marginal expert activation cost for MoE inference.

Principles

• Expert scattering increases memory traffic in MoE speculative decoding.
• MoE verification latency is sensitive to expert overlap and reuse.
• Cost-aware draft selection reduces expert footprint without sacrificing acceptance.

Method

EcoSpec uses a lightweight expert predictor to estimate token-expert activations, a global expert buffer to track covered experts, and a cost-aware selection algorithm to prioritize draft paths balancing acceptance likelihood with marginal expert cost.

In practice

• Achieve up to 1.62x speedup on Qwen3-235B-A22B.
• Reduce HBM read traffic by 11.2 GB/step on Qwen3-235B-A22B.
• Applicable to DeepSeek-V3.1, Qwen3-235B-A22B, GPT-OSS-120B.

Topics

• Mixture-of-Experts
• Speculative Decoding
• LLM Inference
• Memory Optimization
• Expert Routing
• DeepSeek-V3.1

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts — Cost-aware speculative decoding, like EcoSpec, optimizes MoE LLM inference by minimizing expert activation costs during draft selection.
• Why MoE models get more from speculative decoding - Cohere — MoE sparsity and expert routing significantly enhance speculative decoding performance, especially at moderate batch sizes.
• Mixture of Experts (MoE) - More Parameters, Same Compute — Mixture of Experts scales model capacity by having many specialized networks, but keeps per-token compute low by activating only a few.
• Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding — Hybrid Verified Decoding optimizes LLM inference by dynamically selecting between cache and model-based drafting based on predicted verification payoff.
• Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study — On bandwidth-bound edge hardware, MoE inference cost tracks total parameters, not active ones, negating expected efficiency gains.
• Why Half the Experts in an MoE Model May Not Be Needed — ZEDA dynamically reduces MoE inference cost by teaching post-trained models to skip expert computation for predictable tokens via zero-expert self-distillation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.