STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

2026-06-07 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

STAR-KV is an adaptive low-rank KV cache compression framework designed to overcome limitations of prior methods that use fixed or heuristic rank selection, which often struggle with aggressive compression and minimal accuracy degradation. This framework incorporates three key mechanisms: a differentiable thresholding mechanism for optimal rank selection at both attention-head and block levels, a hybrid decomposition strategy that applies different low-rank factorizations based on key and value projection sensitivity, and a low-rank-aware mixed precision quantization leveraging data statistics for near lossless low-bit quantization. Evaluated across multiple Large Language Models and benchmarks, STAR-KV achieves up to 75% KV cache compression and up to 20x overall KV cache reduction when combined with quantization. Enabled by custom Triton-based GPU kernels, it delivers up to 6.9x speedup for the attention module and 3.1x end-to-end generation throughput.

Key takeaway

For Machine Learning Engineers optimizing Large Language Model inference, STAR-KV offers a compelling solution to significantly reduce KV cache memory and boost throughput. You should consider integrating this adaptive low-rank compression framework to achieve up to 75% KV cache compression and 3.1x end-to-end generation speedup, especially when deploying models on resource-constrained hardware. Explore its publicly available code to implement fine-grained rank control and mixed precision quantization.

Key insights

STAR-KV adaptively compresses KV cache using fine-grained rank control, achieving significant memory and speed improvements.

Principles

• Low-rank projection exploits hidden-dimension redundancy.
• Differentiable thresholding optimizes rank selection.
• Hybrid decomposition adapts to projection sensitivity.

Method

STAR-KV employs differentiable thresholding for adaptive rank selection, a hybrid decomposition strategy for key/value projections, and low-rank-aware mixed precision quantization.

In practice

• Achieve up to 75% KV cache compression.
• Utilize Triton-based GPU kernels for attention speedup.
• Combine with quantization for 20x KV cache reduction.

Topics

• KV Cache Compression
• Low-Rank Projection
• Large Language Models
• Mixed Precision Quantization
• Triton GPU Kernels
• Adaptive Rank Control

Code references

• PriyanshBhatnagar/STAR-KV

Best for: MLOps Engineer, AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models — GUI VLM KV cache compression benefits from spatio-temporal adaptive re-weighting, moving beyond fixed saliency and cutoffs.
• Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limit — Exploiting the sequential, predictable nature of KV caches significantly surpasses per-vector compression limits.
• Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation — Moment-KV improves LLM long-generation fidelity by using momentum-driven attention aggregation for decoding-time KV cache compression.
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels — Factored norms and fused kernels significantly optimize high-rank DoRA, reducing memory and improving speed.
• KV Cache Demystified: Speeding Up Large Language Models — KV cache optimizes LLM inference by storing and reusing attention keys and values, preventing redundant computations.
• Still: Amortized KV Cache Compaction in a Single Forward Pass — Still offers an efficient, high-quality KV cache compaction method for long-context language models via a single forward pass Perceiver.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.