NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference

2026-06-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Expert, quick

Summary

NetKV is a novel system designed for disaggregated Large Language Model (LLM) inference that optimizes decode instance selection by considering network topology and dynamic congestion. In disaggregated setups, the KV cache must traverse the datacenter network, directly impacting Time to First Token (TTFT). Existing schedulers primarily focus on compute load and prefix-cache locality, overlooking critical network transfer times. NetKV introduces a thin operator-to-scheduler interface and a network cost oracle, proving that ignoring network terms leads to suboptimal scheduling as context length increases. This O(|D|) per-request greedy algorithm demonstrates robustness to stale telemetry. Evaluated on a 64-GPU four-tier fat-tree simulator using Mooncake traces, NetKV reduced mean TTFT by up to 21.2% compared to round-robin and 17.6% against a tuned cache+load-aware scheduler, while boosting SLO attainment by up to 20.1 percentage points and maintaining Time Between Tokens overhead below 0.5 ms, all without hardware or engine modifications.

Key takeaway

For AI Architects and ML Engineers optimizing disaggregated LLM inference, you should prioritize network-aware scheduling to significantly improve Time to First Token (TTFT). Ignoring network latency in KV cache placement can lead to substantial performance bottlenecks, especially with longer contexts. Implement solutions like NetKV's network cost oracle to factor in topological distance and dynamic congestion, potentially reducing TTFT by over 20% and boosting Service Level Objective attainment without requiring hardware changes.

Key insights

Network-aware scheduling for disaggregated LLM inference significantly reduces TTFT by optimizing KV cache placement.

Principles

• Network topology impacts LLM inference performance.
• Cache-aware scheduling alone is suboptimal for long contexts.
• Robustness to stale telemetry is achievable.

Method

NetKV employs a network cost oracle and an O(|D|) per-request greedy algorithm for decode instance selection, integrating network topology and congestion awareness into scheduling decisions.

In practice

• Integrate network cost into LLM inference schedulers.
• Prioritize TTFT optimization in disaggregated setups.
• Design schedulers robust to network telemetry delays.

Topics

• Disaggregated LLM Inference
• KV Cache Optimization
• Network-Aware Scheduling
• Time to First Token
• Datacenter Networks
• LLM Performance

Best for: MLOps Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer, AI Architect

Related on AIssential

• LLM Inference and Optimization: Fundamentals, Bottlenecks, and Techniques — Optimizing LLM inference requires understanding distinct prefill and decode phases, leveraging KV caching, and applying advanced techniques.
• Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference — Optimizing LLM inference by routing requests to specialized short or long context pools significantly reduces GPU costs and improves reliability.
• Deploying Disaggregated LLM Inference Workloads on Kubernetes — Disaggregated LLM inference optimizes resource use and scaling by separating prefill, decode, and routing stages.
• KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs — KV Packet enables recomputation-free KV cache reuse for LLMs using soft-token adapters and self-supervised distillation.
• ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning — ArborKV optimizes LLM KV cache for tree-based reasoning by aligning eviction with search dynamics, reducing memory by ~4x.
• Streaming Faster Made Our LLM Hub Slower — Batching LLM tokens at the agent level significantly reduces network overhead and improves system stability under load.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.