StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k

2026-05-04 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

DeepSeek-V3.2 and V4 utilize Compressed Sparse Attention (CSA), which involves a learned scoring projection (indexer) to select top-k keys per query for sparse attention. Existing public CSA implementations materialize a large FP32 score tensor, reaching 256 GB for a sequence length of S=65,536 with V4-Flash dimensions, which exceeds single-GPU HBM. StreamIndex is a Triton-based implementation of the CSA pipeline featuring a chunked partition-merge top-k driver that avoids materializing the full intermediate tensor. On an NVIDIA H200, StreamIndex processes V4-Flash dimensions up to S=1,048,576 with a peak HBM usage of 6.21 GB, extending the operational regime by 32x compared to the materialize path which OOMs at S=65,536. StreamIndex maintains bit-exact set-overlap recall at smaller sequence lengths and achieves a mean recall of 1.0000 across various design-space sweeps.

Key takeaway

For AI Engineers developing large language models with sparse attention mechanisms, StreamIndex offers a critical solution to memory limitations. If you are encountering Out-Of-Memory errors when scaling sequence lengths with DeepSeek-V4-like CSA, adopting StreamIndex can extend your operational capacity by 32x, allowing you to process sequences up to S=1,048,576 on a single NVIDIA H200 without sacrificing recall. Consider integrating this Triton implementation to overcome HBM constraints.

Key insights

StreamIndex enables memory-bounded Compressed Sparse Attention by avoiding full score tensor materialization.

Principles

• Chunking prevents OOM errors.
• Sparse attention reduces computation.
• Learned indexers improve efficiency.

Method

StreamIndex uses a chunked partition-merge top-k driver in Triton to process Compressed Sparse Attention scores without materializing the entire intermediate tensor, integrating with TileLang's pipelined attention kernel.

In practice

• Use StreamIndex for large S CSA.
• Integrate with TileLang kernels.
• Optimize chunk and tile sizes.

Topics

• StreamIndex
• Compressed Sparse Attention
• Memory-Bounded Attention
• Streaming Top-k
• Triton

Code references

• RightNow-AI/StreamIndex

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• TritonSigmoid: A fast, padding-aware sigmoid attention kernel for GPUs [R] — TritonSigmoid offers superior performance and stability for variable-length sequence attention, especially in biological modeling.
• Stream-CQSA: Avoiding Out-of-Memory in Attention Computation via Flexible Workload Scheduling — CQS Divide enables exact self-attention decomposition for memory-adaptive, OOM-free LLM processing.
• DeepSeek V4's Secret: 98% Less Memory — Deepseek's new attention design optimizes long context processing by compressing KV caches and selectively attending to relevant tokens.
• StreamKL: Fast and Memory-Efficient KL Divergence for Boosting Attention Distillation — StreamKL enables memory-efficient, long-context attention distillation by eliminating quadratic materialization of attention distributions through an online KL reduction.
• EntmaxKV: Support-Aware Decoding for Entmax Attention — EntmaxKV enables efficient, exact sparse decoding for long-context models by exploiting Entmax's zero-producing property.
• Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents — Vortex simplifies sparse attention algorithm development and deployment by abstracting complex paged memory layouts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.