COBS: Cumulant Order Block Sparse Attention

2026-07-10 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

COBS (Cumulant Order Block Sparse Attention) is a new method designed to improve block sparse attention in large language models, addressing the key-value (KV) cache read bottleneck. While block sparse attention is hardware-friendly, its adoption is limited by the performance of existing selection strategies, such as DeepSeek's Native Sparse Attention (NSA), which rely on first-order approximations of attention mass. COBS enhances NSA by incorporating a novel selector that stores a compressed second-order statistic per block, allowing for more accurate block selection. This approach significantly boosts performance on long-context tasks; on the 32k RULER benchmark, COBS increased the NSA baseline's mean score from 0.2999 to 0.8195, closing approximately 86% of the gap to dense attention's 0.9040. Furthermore, COBS maintains short-context behavior and reduces KV cache read traffic by 15.15x compared to dense attention.

Key takeaway

For Machine Learning Engineers optimizing large language model inference for long-context applications, you should evaluate COBS as a superior alternative to existing block sparse attention methods. This approach significantly improves retrieval performance on benchmarks like 32k RULER, closing 86% of the gap to dense attention, while reducing KV cache read traffic by 15.15x. Implementing COBS allows you to achieve near-dense attention quality and lower negative log-likelihood without compromising short-context behavior, making it a compelling choice for efficient, high-performance LLM deployment.

Key insights

Enhancing block sparse attention by using second-order statistics for more accurate block selection significantly improves long-context performance.

Principles

• Block selection quality dictates sparse attention performance.
• Approximating attention mass is central to efficient selection.
• Higher-order statistics improve attention mass estimation.

Method

COBS extends DeepSeek's NSA by integrating a novel selector that stores a compressed second-order statistic per block, enabling more precise attention mass approximation for block ranking.

In practice

• Achieve 86% of dense attention quality on 32k RULER.
• Reduce KV cache read traffic by 15.15x versus dense.
• Maintain short-context model behavior.

Topics

• Block Sparse Attention
• Large Language Models
• KV Cache Optimization
• Cumulant Expansion
• Long-Context Retrieval
• DeepSeek NSA

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps — Full-attention LLMs are intrinsically sparse and can be efficiently transformed into sparse models with minimal training.
• Multi-Head Latent Attention (MLA) - Explained — Multi-Head Latent Attention reduces KV cache memory by compressing key/value states into a shared latent vector while preserving head diversity.
• Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale — In-context retrieval with LMs is promising but requires attention control for extreme context growth.
• You Only Index Once: Cross-Layer Sparse Attention with Shared Routing — Cross-layer sparse attention (CLSA) shares KV cache and routing indices across layers to improve LLM inference efficiency and quality.
• Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection — Classical data compression provides a parameter-free signal for adaptive sparse attention masks in long sequence models.
• Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation — Semantic segmentation and an efficient distillation framework enable practical, scalable block attention for long-context LLMs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.