CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

2026-06-23 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

CompressKV is a novel KV-cache compression framework designed for resource-efficient long-context Large Language Model (LLM) inference, specifically targeting GQA-based architectures. It addresses the memory and decoding cost bottlenecks of KV caches on constrained hardware, which existing heuristic eviction methods often exacerbate by degrading performance. CompressKV distinguishes itself by identifying "Semantic Retrieval Heads" (SRHs) that capture critical initial, final, and semantically important mid-context tokens. These SRHs guide the selection of KV pairs to retain, rather than aggregating scores from all attention heads. Furthermore, the framework allocates cache budgets across layers based on offline eviction error estimates. Experiments on LongBench and Needle-in-a-Haystack demonstrate its effectiveness, preserving over 97% of full-cache performance with only 3% KV cache on LongBench QA tasks and achieving 90% accuracy with 0.7% KV storage on Needle-in-a-Haystack.

Key takeaway

For AI Engineers optimizing long-context LLM deployments on resource-constrained hardware, consider implementing CompressKV's semantic-retrieval-guided KV-cache compression. This approach significantly reduces memory footprint while maintaining high performance, as demonstrated by preserving over 97% accuracy with only 3% KV cache usage. You should explore integrating this method to improve throughput and reduce operational costs for your GQA-based LLMs.

Key insights

CompressKV uses Semantic Retrieval Heads and layer-wise budget allocation to efficiently compress KV caches for long-context LLMs.

Principles

• Attention heads exhibit distinct functionalities for token importance.
• Layer-wise cache budget allocation optimizes eviction efficiency.

Method

CompressKV identifies Semantic Retrieval Heads (SRHs) to select critical tokens for retention, then allocates cache budgets across layers using offline eviction error estimates.

In practice

• Achieve >97% performance with 3% KV cache on LongBench.
• Attain 90% accuracy using 0.7% KV storage on Needle-in-a-Haystack.

Topics

• KV Cache Compression
• Long-Context LLMs
• Semantic Retrieval
• GQA Architectures
• LLM Inference Optimization
• Memory Footprint Reduction

Code references

• TUDa-HWAI/CompressKV
• CURRENTF/Sparse-vLLM
• w-yibo/R1-Compress

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference — CompressKV improves long-context LLM inference efficiency by selectively compressing KV caches using Semantic Retrieval Heads and layer-wise budget allocation.
• PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression — PolyKV optimizes KV cache compression by applying heterogeneous, layer-wise policies and non-uniform budget allocation, significantly improving long-context LLM performance.
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression — OjaKV dynamically adapts KV cache compression using online PCA and a hybrid storage policy to maintain accuracy in long-context LLMs.
• The Complete Guide to Inference Caching in LLMs — Inference caching optimizes LLM performance and cost by reusing computation results across three distinct layers.
• Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention — LLM architectures are evolving with complex, targeted tweaks to optimize long-context efficiency and reduce computational overhead.
• End-to-End Context Compression at Scale — Encoder-decoder compression can surpass KV cache methods for efficient, high-quality long-context LLM inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.