Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Expert, extended

Summary

This survey, "Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization," by Jiang et al., systematically reviews methods for optimizing the key-value (KV) cache in large language model (LLM) serving systems. It addresses the memory-intensive and costly nature of LLM inference by focusing on system-aware KV infrastructure (sKis). The authors categorize existing optimization efforts into three dimensions: temporal (execution and scheduling), spatial (placement and migration), and structural (representation and retention). The work provides a novel behavior-oriented taxonomy, analyzing cross-behavior co-design patterns and behavior-objective links. It identifies critical open challenges, including the need for SLO-driven tail control, energy-aware sKis, trustworthy sKis, generalizable hardware-aware execution and compute device orchestration, co-optimization, and unified benchmarks for this rapidly evolving field.

Key takeaway

For Machine Learning Engineers or AI Architects designing LLM serving systems, you must holistically optimize KV cache infrastructure across temporal, spatial, and structural dimensions. Focus on co-designing these behaviors, integrating SLO-driven tail control, and addressing energy and trustworthiness concerns. Prioritize unified benchmarks for reliable comparisons, as isolated optimizations often fail to translate into real-world system gains. Your strategy should account for tail latency and quality robustness, especially under bursty or long-context workloads.

Key insights

The KV cache is central to efficient LLM serving, requiring system-aware optimization across temporal, spatial, and structural dimensions.

Principles

• KV cache optimization is a multi-objective problem.
• Outliers significantly impact low-bitwidth quantization.
• System co-design is crucial for KV cache compression gains.

Method

The survey categorizes KV cache optimization methods into temporal (execution/scheduling), spatial (placement/migration), and structural (representation/retention) dimensions, analyzing cross-behavior co-design and objective links.

In practice

• Prioritize KV usage in scheduling for latency-sensitive workloads.
• Keep hot KV caches close to compute devices.
• Isolate outliers in KV cache quantization for better quality.

Topics

• KV Cache Optimization
• LLM Serving Systems
• Memory Management
• Quantization
• Scheduling Algorithms
• Trustworthy AI

Code references

• jjiantong/Awesome-KV-Cache-Optimization
• NVIDIA/TensorRT-LLM
• ray-project/llmperf

Best for: Research Scientist, MLOps Engineer, AI Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization — System-aware KV cache optimization for LLM serving is categorized into temporal, spatial, and structural dimensions to enhance efficiency.
• Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving — KV-cache optimization effectiveness is workload-dependent, with stable quantization balancing quality and performance.
• Kalypso: Relational LLM Serving — Relational LLM serving reuses KV-cache state through query-aware pipelining, significantly boosting semantic query execution efficiency.
• The KV Cache Is Just Memoization — The KV Cache optimizes LLM generation by memoizing attention keys and values, reducing computational complexity from quadratic to linear.
• [Framework] The Asymmetric Key-Value Cache Compression — Asymmetric, co-designed quantization of KV caches and weights is crucial for efficient LLM context scaling.
• RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention — RedKnot optimizes LLM serving by managing KV cache at the head level, recognizing varied utility across attention heads.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.