SpecMD: A Comprehensive Study on Speculative Expert Prefetching

2026-05-06 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

SpecMD is a new standardized framework designed for benchmarking Mixture-of-Experts (MoE) model caching policies across diverse hardware configurations. Developed to address the limited understanding of how various caching policies interact with different hardware specifications, SpecMD enables exhaustive benchmarking of MoE caching strategies. Through its use, researchers reproduced and extended prior approaches under controlled, realistic constraints. Experiments conducted with SpecMD revealed that MoE expert access patterns do not align with traditional temporal locality assumptions, such as those used by LRU or LFU policies. This observation led to the proposal of Least-Stale, a novel eviction policy that leverages MoE's predictable expert access to achieve up to an 85x reduction in collision misses compared to LRU. These improvements result in over 88% hit rates and up to a 34.7% Time-to-first-token (TTFT) reduction on OLMoE, utilizing only 5% or 0.6GB of VRAM cache capacity.

Key takeaway

For AI Engineers optimizing MoE model inference, understanding that traditional caching policies like LRU are inefficient for MoE expert access is critical. You should investigate implementing the Least-Stale eviction policy, which leverages predictable access patterns to significantly reduce collision misses and improve Time-to-first-token (TTFT) performance, even with minimal VRAM cache capacity.

Key insights

SpecMD benchmarks MoE caching, revealing non-temporal expert access and enabling the efficient Least-Stale eviction policy.

Principles

• MoE expert access lacks temporal locality.
• Predictable access patterns can optimize caching.

Method

SpecMD provides a standardized framework for benchmarking ad-hoc MoE cache policies on various hardware, allowing for controlled reproduction and extension of prior approaches under realistic constraints.

In practice

• Implement Least-Stale for MoE caching.
• Reduce VRAM cache capacity to 5% for OLMoE.

Topics

• Mixture-of-Experts
• SpecMD Framework
• Expert Caching
• Least-Stale Policy
• Time-to-first-token

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference — Expert requests in MoE LLMs exhibit strong spatio-temporal correlations, enabling predictable prefetching for efficiency.
• SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents — SpecHop accelerates multi-hop LLM tool use by continuously speculating future actions and verifying them asynchronously.
• Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching — Semantic Cache Distillation efficiently transfers LLM KV caches using compact semantic codes, significantly speeding up time-to-first-token while preserving quality.
• Why MoE models get more from speculative decoding - Cohere — MoE sparsity and expert routing significantly enhance speculative decoding performance, especially at moderate batch sizes.
• The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference — FP16 KV cache inference is fundamentally non-equivalent to recomputation due to non-associativity, causing deterministic token divergence.
• LLM Inference and Optimization: Fundamentals, Bottlenecks, and Techniques — Optimizing LLM inference requires understanding distinct prefill and decode phases, leveraging KV caching, and applying advanced techniques.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.