REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression

2025-02-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

The SurfaceLogicKV method addresses the significant memory pressure of KV cache storage in Large Language Models (LLMs) for long-context inference. It proposes a novel two-stage compression approach based on distinguishing attention behaviors into "surface memorization" and "logic construction." Observations on Llama-3-8B-Instruct show that approximately 98.5% of attention heads ignore irrelevant information, with 1.5% engaging in logic construction and 0.5% in surface memorization. SurfaceLogicKV integrates these layer- and head-wise behaviors to dynamically allocate KV cache budgets. Experiments on Llama-3-8B-Instruct, Mistral-7B-Instruct, and 123B Mistral-Large-Instruct-2411 across benchmarks like LongBench and LooGLE, with context lengths up to 129K, demonstrate improved compression robustness and competitive performance, sometimes even surpassing FullKV, particularly at a β=1.351 allocation ratio.

Key takeaway

For Machine Learning Engineers optimizing LLM inference for long contexts, you should consider adopting attention behavior-driven KV cache compression methods like SurfaceLogicKV. By analyzing and allocating resources based on "surface memorization" and "logic construction" behaviors, you can achieve more robust performance and memory efficiency than traditional token- or layer-wise approaches. This allows for maintaining competitive inference quality even with significantly reduced KV cache sizes, improving deployment feasibility for large models.

Key insights

LLM KV cache compression can be robustly optimized by distinguishing and allocating resources based on "surface memorization" and "logic construction" attention behaviors.

Principles

• Attention heads exhibit distinct behaviors: surface memorization (0.5%), logic construction (1.5%), and irrelevant information ignoring (98.5%).
• Attention behaviors vary significantly across different layers and heads within an LLM.
• A minimum KV cache budget per layer/head enhances robustness against over-pruning.

Method

SurfaceLogicKV calculates Surface Memorization Score ($SF_{sc}$) and Logic Construction Score ($LG_{sc}$) to derive a comprehensive Inference Score ($INF_{sc}$). This $INF_{sc}$ guides a two-stage, layer- and head-wise dynamic KV budget allocation, combining fixed and dynamic portions.

In practice

• Analyze attention weight distributions using Needle-In-A-Haystack to identify behavioral patterns.
• Implement a hybrid fixed and dynamic KV budget allocation strategy for LLM inference.

Topics

• KV Cache Compression
• Large Language Models
• Attention Mechanisms
• SurfaceLogicKV
• Needle-In-A-Haystack
• Inference Optimization

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference — CompressKV improves long-context LLM inference efficiency by selectively compressing KV caches using Semantic Retrieval Heads and layer-wise budget allocation.
• The KV Cache Is Just Memoization — The KV Cache optimizes LLM generation by memoizing attention keys and values, reducing computational complexity from quadratic to linear.
• A Visual Guide to Attention Variants in Modern LLMs — Modern LLM architectures prioritize efficiency and long-context handling through diverse attention mechanisms and hybrid designs.
• A brief history of KV cache compression developments — Mathematical advancements in KV cache compression have dramatically expanded LLM context windows, outperforming hardware memory growth.
• Information-Aware KV Cache Compression for Long Reasoning — InfoKV improves LLM long reasoning by combining information-theoretic signals with attention for KV cache compression.
• A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs — KV cache compression is near-lossless at 2–3× by treating it as a tensor and jointly allocating ranks and residual bits.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.