KVpop -- Key-Value Cache Compression with Predictive Online Pruning

2026-07-06 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

KVpop is a novel method addressing the key-value (KV) cache growth bottleneck in autoregressive decoding, where memory and bandwidth scale linearly with context length. Unlike existing eviction methods that use static heuristics or proxy scores, KVpop learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. Its scorer is trained against a new future-attention target, computed efficiently without materializing dense attention maps. KVpop also incorporates a delayed memory-based scorer that defers scoring to exploit near-future context. Benchmarked on AIME and HMMT mathematical reasoning, KVpop on Qwen3-4B retains 98% of full-attention performance at 75% KV cache compression and 97% at 88% compression, consistently outperforming baselines. Qwen3-8B shows even stronger results, achieving near-full teacher performance.

Key takeaway

For Machine Learning Engineers optimizing large language model inference, KVpop offers a compelling strategy to reduce memory footprint without sacrificing performance. If you are struggling with KV cache growth, consider implementing a learned eviction policy supervised by future-attention signals. This approach allows you to achieve significant compression, like 75% on Qwen3-4B, while maintaining near-full attention quality, directly addressing memory and bandwidth bottlenecks in autoregressive decoding.

Key insights

KVpop uses future-attention signals to learn KV cache eviction, significantly compressing memory while preserving model quality.

Principles

• Direct supervision improves KV cache eviction.
• Future-attention targets enhance utility tracking.
• Delayed scoring exploits near-future context.

Method

KVpop trains a scorer against a novel future-attention target for keep-or-drop decisions, efficiently computed without dense attention maps, and employs a delayed memory-based scorer.

In practice

• Achieve 75% KV cache compression on Qwen3-4B.
• Maintain 98% full-attention performance.
• Outperform static eviction baselines.

Topics

• KV Cache Compression
• Autoregressive Decoding
• Predictive Pruning
• Future Attention
• Qwen3 Models
• LLM Inference Optimization

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• KV Caching in LLMs: A Guide for Developers — KV caching reuses attention's key and value projections to accelerate autoregressive LLM inference by avoiding redundant computation.
• STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control — STAR-KV adaptively compresses KV cache using fine-grained rank control, achieving significant memory and speed improvements.
• Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limit — Exploiting the sequential, predictable nature of KV caches significantly surpasses per-vector compression limits.
• Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation — Moment-KV improves LLM long-generation fidelity by using momentum-driven attention aggregation for decoding-time KV cache compression.
• Fast KVzip: Much Faster KV Cache Eviction for Cheaper Inference — Causal attention in decoder-only LLMs limits context utilization when options precede context.
• Epiphany-Aware KV Cache Eviction Without the Attention Matrix — EpiKV efficiently evicts KV cache tokens by scoring internal model representation changes, bypassing attention matrix computation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.