A brief history of KV cache compression developments

2026-06-15 · Source: Martin Alderson · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

KV cache compression has achieved a remarkable 100x reduction in memory needed per token since 2017, significantly outpacing the 18x improvement in GPU memory (from 16GB to 288GB). This efficiency is crucial for Large Language Models (LLMs), as the KV cache stores conversation context, directly impacting session length. Initially, a 128K token context required ~340GB of GPU memory with MHA in 2017. Key advancements include MQA (2019) offering a 64x reduction but with quality loss, and GQA (2023) providing an 8x reduction with minimal quality impact, quickly adopted by models like Llama 2 70B and Mistral. Sliding window attention also emerged in 2023. DeepSeek's MLA (2024) achieved a 93% reduction by compressing keys and values into latent vectors, even improving quality. Further gains came from 8-bit or 4-bit KV cache quantization. More recently, linear-attention hybrids like those in Qwen3-Next and Kimi Linear (2025) enabled 1M token context windows by using fixed-size states for most layers, leading to models like Anthropic's 1M context Claude.

Key takeaway

For AI Scientists and Machine Learning Engineers building LLM-powered agents, understanding KV cache compression is critical. Your choice of model architecture and inference techniques directly impacts context window capacity and operational costs. Prioritize models incorporating GQA, MLA, or linear-attention hybrids, and implement KV cache quantization to maximize context length and enable more sophisticated agentic workflows without excessive memory overhead. This allows for richer interactions and document processing.

Key insights

Mathematical advancements in KV cache compression have dramatically expanded LLM context windows, outperforming hardware memory growth.

Principles

• Aggressive compression often sacrifices quality.
• Hybrid approaches balance efficiency and quality.
• Efficiency gains often fund new capabilities.

Method

KV cache compression evolved from sharing KV heads (MQA, GQA) to latent vector compression (MLA) and fixed-size state linear-attention hybrids, often combined with quantization.

In practice

• Implement GQA for balanced memory savings.
• Explore MLA for significant cache reduction.
• Utilize 8-bit/4-bit KV cache quantization.

Topics

• KV Cache Compression
• Large Language Models
• Attention Mechanisms
• GPU Memory Optimization
• Agentic AI
• Quantization

Best for: MLOps Engineer, Research Scientist, AI Architect, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression — LLM KV cache compression can be robustly optimized by distinguishing and allocating resources based on "surface memorization" and "logic construction" attention behaviors.
• Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention — LLM architectures are evolving with complex, targeted tweaks to optimize long-context efficiency and reduce computational overhead.
• Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization — The KV cache is central to efficient LLM serving, requiring system-aware optimization across temporal, spatial, and structural dimensions.
• Rotate, Then Round: The Geometry of KV-Cache Compression — KV-cache compression evolved from simple rounding to geometric transformations, but production revealed that unbiasedness isn't enough when softmax amplifies variance.
• The KV Cache Is Just Memoization — The KV Cache optimizes LLM generation by memoizing attention keys and values, reducing computational complexity from quadratic to linear.
• CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference — CompressKV uses Semantic Retrieval Heads and layer-wise budget allocation to efficiently compress KV caches for long-context LLMs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Martin Alderson.