KVpop -- Key-Value Cache Compression with Predictive Online Pruning

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

KVpop (KV compression with Predictive Online Pruning) is a novel sparse-attention retrofit designed to address the KV cache bottleneck in Transformer-based large language models during long-context inference. It maintains a fixed per-head KV budget by retaining sink tokens, a protected window of recent tokens, and a learned long-range top-k cache for older tokens. KVpop distinguishes itself by supervising its lightweight head-wise scoring modules with future-attention mass, evaluated at the eviction boundary, and efficiently computed via a transposed-attention pass. It also supports stateful scorers that delay scoring until the eviction boundary, integrating near-future context. Empirically, KVpop preserves 97% and 100% of dense-attention performance on Qwen3-4B and Qwen3-8B, respectively, at 88% KV cache compression, outperforming heuristic and learned eviction baselines like DMS. It also demonstrates strong generalization to out-of-domain tasks and significantly reduces VRAM growth, maintaining constant memory use and faster long-generation decoding compared to dense attention and DMS.

Key takeaway

For AI Scientists and Machine Learning Engineers optimizing LLM inference for long contexts, KVpop offers a robust solution to the KV cache memory bottleneck. You should consider retrofitting your Transformer models with KVpop to achieve significant memory compression (e.g., 88%) while preserving high performance (e.g., 97-100% on Qwen3 models). This approach ensures stable, faster decoding for long generations and generalizes well across diverse reasoning tasks, making it a strong candidate for deploying efficient, long-context LLMs.

Key insights

KVpop uses future-attention supervision and delayed stateful scoring to predict token utility for efficient, fixed-budget KV cache compression.

Principles

• Supervise eviction decisions at the boundary where tokens become evictable.
• Delay scoring to integrate near-future context for better utility prediction.
• Homogeneous per-head cache budgets enable efficient GPU execution.

Method

KVpop trains lightweight scorers using a boundary-aware loss and a transposed-attention pass for future-attention targets. It employs a Fenwick tree and FlexAttention for running top-k sparse mask generation.

In practice

• Apply KVpop as a post-training retrofit for Transformer LLMs.
• Use mLSTM-based stateful scorers for improved token accuracy.
• Prioritize retaining reasoning-structural tokens over purely numeric ones.

Topics

• KV Cache Compression
• Transformer LLMs
• Sparse Attention
• Predictive Pruning
• mLSTM Scorers
• Long-Context Inference

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• KVpop -- Key-Value Cache Compression with Predictive Online Pruning — KVpop uses future-attention signals to learn KV cache eviction, significantly compressing memory while preserving model quality.
• A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs — JoLT compresses LLM KV caches by treating them as third-order tensors, combining Tucker decomposition with JL-residual allocation for near-lossless results.
• KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling — KV-PRM uses KV-cache transfer to drastically reduce computational cost for process reward modeling in multi-agent LLM systems.
• The KV Cache Is Just Memoization — The KV Cache optimizes LLM generation by memoizing attention keys and values, reducing computational complexity from quadratic to linear.
• VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs — Variable-rate rank allocation by query salience robustly compresses KV caches without training, preserving accuracy better than eviction.
• Fast KVzip: Much Faster KV Cache Eviction for Cheaper Inference — Causal attention in decoder-only LLMs limits context utilization when options precede context.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.