HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference

2026-07-13 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Expert, quick

Summary

HCRMap is a hot expert residency mapping framework designed to optimize Mixture-of-Experts (MoE) Large Language Model (LLM) inference on 3.5D multi-chiplet systems. It addresses the persistent expert hotness skew where a small set of experts receives most tokens, causing compute imbalance and amplifying pressure on communication, memory bandwidth, I/O, and execution queues. HCRMap dynamically determines which experts to promote, retain, demote, or evict based on expert hotness, weight loading cost, migration overhead, and runtime resource pressure. It then maps routed token groups to suitable resident replicas, jointly mitigating various bottlenecks. Experimental results demonstrate HCRMap reduces end-to-end latency by 43.6% and 43.0% over Hydra in prefill and decode stages, respectively; by 34.5% and 33.1% over MoEntwine; and by 46.7% and 46.0% over PIMoE.

Key takeaway

For machine learning engineers deploying MoE LLMs on 3.5D multi-chiplet systems, you should consider HCRMap's dynamic expert replica management to significantly reduce inference latency and alleviate communication, memory, and queue bottlenecks. This approach offers substantial performance gains over existing methods like Hydra, MoEntwine, and PIMoE, making it critical for optimizing resource utilization in high-performance MoE deployments.

Key insights

HCRMap dynamically manages hot expert replicas to mitigate resource bottlenecks in MoE LLM inference on 3.5D chiplets.

Principles

• Expert hotness skew amplifies resource pressure.
• Dynamic replica placement reduces bottlenecks.
• Optimizing expert residency improves MoE inference.

Method

HCRMap dynamically promotes, retains, demotes, or evicts experts based on hotness, loading cost, migration overhead, and runtime resource pressure, then maps token groups to replicas.

In practice

• Reduce MoE LLM inference latency.
• Optimize 3.5D multi-chiplet utilization.
• Manage expert replicas efficiently.

Topics

• Mixture-of-Experts
• Large Language Models
• 3.5D Chiplets
• Inference Optimization
• Expert Residency Mapping
• Resource Management
• Latency Reduction

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference — HCRMap dynamically manages hot expert replicas in 3.5D MoE systems to mitigate resource bottlenecks caused by expert hotness skew.
• Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study — On bandwidth-bound edge hardware, MoE inference cost tracks total parameters, not active ones, negating expected efficiency gains.
• SpecMD: A Comprehensive Study on Speculative Expert Prefetching — SpecMD benchmarks MoE caching, revealing non-temporal expert access and enabling the efficient Least-Stale eviction policy.
• FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs — FoMoE partitions MoE expert layers across workers to overcome full-replica limitations in distributed LLM training.
• MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers — MoECa accelerates DiT-MoE inference by reusing expert-branch features across timesteps, achieving up to 2.83x speedup.
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0 — MLPerf Training v6.0 now benchmarks MoE LLMs, reflecting industry shifts to sparse computation and specialized architectures.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.