EntmaxKV: Support-Aware Decoding for Entmax Attention

2026-05-23 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

EntmaxKV is a novel sparse decoding framework designed to mitigate KV-cache memory traffic limitations in long-context autoregressive decoding, a problem exacerbated by cache size growing linearly with context length. Unlike softmax attention, which has dense tails and discards probability mass when truncated, Entmax attention generates exact zeros, transforming sparse decoding into a support recovery task. EntmaxKV leverages this property by exploiting sparsity *before* KV pages are loaded, integrating query-aware page scoring, support-aware candidate selection, and sparse entmax attention. The framework includes a Gaussian-aware entmax selector that estimates the entmax threshold from lightweight page statistics. Empirically, EntmaxKV demonstrates superior performance over softmax-based sparse decoding, dropping less probability mass and achieving lower output error. It closely matches full-cache entmax performance while using a small fraction of the KV cache, yielding up to 3.36x speedup over softmax and 5.43x over entmax full attention baselines at 1M context length.

Key takeaway

For Machine Learning Engineers optimizing large language models for long-context inference, EntmaxKV offers a significant solution to KV-cache memory bottlenecks. You should consider integrating this framework to achieve substantial speedups, up to 5.43x, while maintaining accuracy comparable to full-cache entmax. This approach allows you to reduce memory traffic and computational overhead, making long-context applications more feasible and cost-effective without sacrificing model performance.

Key insights

EntmaxKV enables efficient, exact sparse decoding for long-context models by exploiting Entmax's zero-producing property.

Principles

• Entmax attention allows exact sparse decoding.
• Exploit sparsity before KV-cache loading.
• Truncation error is controlled by dropped probability mass.

Method

EntmaxKV combines query-aware page scoring, support-aware candidate selection, and sparse entmax attention, using a Gaussian-aware selector to adapt budget.

In practice

• Implement EntmaxKV for long-context LLMs.
• Use Gaussian-aware selector for dynamic budget.
• Evaluate against full-cache entmax for performance.

Topics

• Entmax Attention
• Sparse Decoding
• KV-Cache Optimization
• Long-Context LLMs
• Memory Efficiency
• Autoregressive Decoding

Code references

• deep-spin/entmaxkv

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing — DASH-KV accelerates long-context LLM inference by reformulating attention as asymmetric deep hashing, achieving linear complexity.
• End-to-End Context Compression at Scale — Encoder-decoder compression can surpass KV cache methods for efficient, high-quality long-context LLM inference.
• You Only Index Once: Cross-Layer Sparse Attention with Shared Routing — Cross-layer sparse attention (CLSA) shares KV cache and routing indices across layers to improve LLM inference efficiency and quality.
• Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limit — Exploiting the sequential, predictable nature of KV caches significantly surpasses per-vector compression limits.
• CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference — CompressKV improves long-context LLM inference efficiency by selectively compressing KV caches using Semantic Retrieval Heads and layer-wise budget allocation.
• KV Cache Demystified: Speeding Up Large Language Models — KV cache optimizes LLM inference by storing and reusing attention keys and values, preventing redundant computations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.