PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

2026-07-17 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

PagedWeight is a novel management method designed for serving Mixture-of-Experts (MoE) large language models (LLMs), specifically addressing the tension between GPU memory demands for model weights and growing KV cache sizes in intensive serving scenarios. Proposed on 2026-07-17, this system dynamically quantizes MoE model weights at runtime, effectively balancing expert-weight precision with KV cache requirements. PagedWeight navigates the complex tradeoff among task accuracy, memory consumption, and throughput/latency. It achieves FP16-equivalent accuracy with significant GPU memory savings of up to 72.0% and a 1.94x throughput improvement. Furthermore, PagedWeight improves quality over existing quantization methods by up to 39.3% at a similar memory budget, incurring at most a 4.1% throughput loss.

Key takeaway

For MLOps Engineers deploying Mixture-of-Experts LLMs, PagedWeight offers a critical solution for memory and throughput challenges. You should evaluate this dynamic quantization approach. It achieves FP16-equivalent accuracy while reducing GPU memory consumption by up to 72.0% and boosting throughput by 1.94x. This allows you to serve larger models or more concurrent requests on existing hardware, significantly improving cost-efficiency and scalability.

Key insights

PagedWeight dynamically quantizes MoE LLM weights at runtime to balance accuracy, memory, and throughput for efficient serving.

Principles

• MoE LLM serving faces KV cache vs. weight memory tension.
• Dynamic quantization can balance precision and memory.
• Optimizing MoE serving requires navigating accuracy, memory, throughput tradeoffs.

Method

PagedWeight dynamically quantizes MoE model weights at runtime, balancing expert-weight precision with KV cache sizes to optimize task accuracy, memory consumption, and throughput/latency.

In practice

• Achieve FP16 accuracy with 72.0% GPU memory savings.
• Improve throughput by 1.94x for MoE LLMs.
• Enhance quality by 39.3% at similar memory budgets.

Topics

• Mixture-of-Experts
• LLM Serving
• Dynamic Quantization
• GPU Memory Optimization
• KV Cache
• Model Quantization

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Optimize, deploy, and benchmark an open-source LLM with vLLM — Efficient LLM inference is achieved through specialized memory management and serving systems like vLLM.
• Mixture of Experts (MoEs) in Transformers — MoEs offer high model capacity with reduced inference costs by activating only a subset of experts per token.
• CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation — Disaggregating KV-cache and FFN weights into separate GPU pools optimizes multi-LLM serving for cold MoE models.
• Qwen3.5 9B MoQ: Inside a Strong 3.6-bit GGUF — Advanced quantization and MoE architectures significantly boost LLM efficiency and performance on diverse hardware.
• PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs — Adaptive page-aware decode scheduling significantly boosts long-context LLM serving throughput on commodity GPUs by optimizing KV cache movement.
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0 — MLPerf Training v6.0 now benchmarks MoE LLMs, reflecting industry shifts to sparse computation and specialized architectures.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.