Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Expert, quick

Summary

A proactive prefill-deflecting scheduler for disaggregated LLM serving addresses the common issue of prefill node saturation and underutilized decode nodes under bursty workloads. In a production-style A100 cluster with 2 prefill and 2 decode nodes (2P2D), 77-98% of P95 Time-to-First-Token (TTFT) is attributed to queuing and inter-node KV-cache transfer, with prefill execution accounting for only 2-23%. The proposed scheduler enables decode nodes to serve prefill phases of requests as chunked steps, interleaved with their in-flight decode batches. It estimates TTFT on prefill nodes and identifies optimal chunk schedules on decode nodes that maintain Time-Between-Tokens (TBT) SLOs for existing decodes. Deflection occurs if the decode path improves tail latency, eliminating inter-node KV transfer. Implemented on vLLM and evaluated with DeepSeek-V2-Lite on production traces, this approach reduces P95 TTFT by up to 81% and raises SLO attainment by up to 79%, incurring a sub-millisecond routing cost.

Key takeaway

For MLOps Engineers managing disaggregated LLM serving performance, especially under bursty workloads, your current disaggregated LLM serving setup likely suffers from high P95 TTFT due to prefill node saturation and KV-cache transfer. Implement a prefill-deflecting scheduler to offload prefill tasks to underutilized decode nodes. This strategy can significantly reduce tail latency and improve service level objective attainment by up to 79%, while minimizing inter-node data transfer.

Key insights

Disaggregated LLM serving benefits from deflecting prefill tasks to underutilized decode nodes to reduce tail latency and KV-cache transfer.

Principles

• Prefill phase often bottlenecks disaggregated LLM serving.
• Interleaving prefill with decode batches improves efficiency.
• Proactive scheduling can mitigate tail latency.

Method

Estimate request TTFT on prefill nodes. Search decode nodes for largest chunk schedule maintaining TBT SLOs. Deflect prefill to decode if it improves tail latency, eliminating inter-node KV transfer.

In practice

• Implement prefill deflection in vLLM-based systems.
• Monitor P95 TTFT and SLO attainment for improvements.
• Consider chunked prefill for bursty LLM workloads.

Topics

• Disaggregated LLM Serving
• Prefill Deflection
• Tail Latency Optimization
• vLLM
• DeepSeek-V2-Lite
• GPU Scheduling

Best for: AI Scientist, Research Scientist, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• A Deep Dive Into Disaggregation in LLMs — Disaggregated LLM inference optimizes performance and cost by separating prefill, decode, and routing stages.
• Service-Induced Congestion in Memory-Constrained LLM Serving — LLM serving's memory growth causes "service-induced congestion," leading to instability and significant throughput loss, which heterogeneity can mitigate.
• Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption — Multi-model LLM serving requires schedulers to account for non-linear offloading sensitivity and high preemption costs across diverse models and hardware.
• Article: Stragglers, Not Failures: How Adaptive Hedged Requests Reduce p99 Latency by 74 Percent — Adaptive hedging, using real-time latency distribution learning and a budget, effectively mitigates straggler-induced tail latency without manual tuning.
• I Deployed Local LLMs in Production for a Year. Part 1: The Mental Model — Production LLM deployment requires understanding hidden queues, KV cache memory, and multi-phase model loading beyond basic tutorials.
• Clarifai 12.3: Introducing KV Cache-Aware Routing — Intelligent routing and caching optimize LLM inference by leveraging KV cache state and pre-warmed infrastructure.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.