DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

DepthWeave-KV is a novel token-adaptive cache compression method designed to address memory bandwidth and capacity limitations in long-context language model inference. It factorizes key and value states across neighboring transformer layers using shared low-rank channel bases, while preserving lightweight token-specific residuals for sensitive attention behaviors. The system integrates cross-depth residual factorization with a token-conditional depth router, which intelligently allocates higher reconstruction rank to critical tokens like instructions. Furthermore, DepthWeave-KV employs calibration-free online error tracking via attention-output probes, enabling adaptive compression during generation without requiring base model retraining. A fused CUDA implementation optimizes decode-time memory traffic by jointly performing basis lookup, residual dequantization, and attention projection. Benchmarked across LongBench, Needle-in-a-Haystack, L-Eval, and long-form QA/summarization tasks, DepthWeave-KV achieves near-full-cache quality with significantly reduced memory, demonstrating an 8.3x KV memory reduction and processing 72.8 tokens per second at 64K context, outperforming prior compressed caches in average score and retrieval accuracy.

Key takeaway

For Machine Learning Engineers optimizing long-context LLMs, DepthWeave-KV offers a critical solution to memory bottlenecks. You can achieve an 8.3x KV memory reduction and maintain near-full-cache task quality, even at 64K context lengths. This method allows you to deploy larger models or longer contexts on existing hardware, significantly improving inference efficiency and retrieval accuracy without retraining your base models. Consider integrating this approach to enhance your LLM deployment capabilities.

Key insights

Token-adaptive cross-layer residual factorization compresses KV caches, improving long-context LLM inference efficiency and quality without retraining.

Principles

• Adaptive compression improves retrieval.
• Factorize states across transformer layers.
• Prioritize critical tokens for higher rank.

Method

DepthWeave-KV factorizes KV states across layers using shared low-rank bases, applies token-conditional depth routing for critical tokens, and uses online error tracking from attention-output probes for adaptive compression.

In practice

• Achieve 8.3x KV memory reduction.
• Maintain quality at 64K context.
• Improve retrieval accuracy.

Topics

• KV Cache Compression
• Long-Context LLMs
• Transformer Architectures
• Adaptive Compression
• Memory Optimization
• CUDA Acceleration

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference — CompressKV improves long-context LLM inference efficiency by selectively compressing KV caches using Semantic Retrieval Heads and layer-wise budget allocation.
• KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing — KVEraser efficiently erases LLM context by replacing KV states with learned steering states, avoiding costly full recomputation.
• WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering — WaveFilter uses wavelet transform to identify critical tokens in long sequences, creating a sparse KV Cache to improve Diffusion LLM performance.
• Towards Selective Depth Mixing: Attention Residuals for Stable, Reusable Features in Deep Transformers — Novel architectural and inference techniques enhance LLM efficiency and performance by optimizing residual connections, KV cache management, and sampling.
• Still: Amortized KV Cache Compaction in a Single Forward Pass — Still offers an efficient, high-quality KV cache compaction method for long-context language models via a single forward pass Perceiver.
• End-to-End Context Compression at Scale — Encoder-decoder compression can surpass KV cache methods for efficient, high-quality long-context LLM inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.