Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

2026-07-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Sparse Delta Memory (SDM) is a new architecture designed to significantly scale the hidden state capacity of gated linear Recurrent Neural Networks (RNNs). While linear attention models offer fixed state size and compute per token, they typically underperform softmax-attention-based transformers in long-context recall due to limited state size. SDM addresses this by extending the Gated DeltaNet architecture, replacing its dense key-value outer product with a sparse addressing scheme for reads and writes to a large explicit memory. This approach allows for orders of magnitude higher memory capacity. The research demonstrates that, under an isoFLOP constraint and with an identical number of parameters, increased state memory capacity substantially improves performance on in-context learning and long-context retrieval tasks. Furthermore, learning the initial state of the SDM memory as a parametric memory enhances performance across a wide range of common-knowledge and reasoning tasks.

Key takeaway

For Machine Learning Engineers developing long-context models, Sparse Delta Memory (SDM) offers a path to overcome linear RNN state limitations. If you are struggling with recall in linear attention architectures, consider implementing SDM's sparse memory scheme. This approach significantly boosts performance on in-context learning and retrieval tasks, even under isoFLOP constraints. Additionally, using SDM's parametric memory can enhance your model's common-knowledge and reasoning capabilities.

Key insights

Sparse Delta Memory scales linear RNNs' state for superior long-context and reasoning performance.

Principles

• Sparse memory addressing scales RNN state.
• Higher state capacity improves long-context recall.
• Parametric memory enhances reasoning tasks.

Method

SDM extends Gated DeltaNet by replacing dense key-value outer products with sparse reads and writes to a large explicit memory, enabling orders of magnitude higher state capacity.

In practice

• Improve long-context recall in linear RNNs.
• Enhance common-knowledge reasoning tasks.

Topics

• Sparse Delta Memory
• Linear RNNs
• Long-Context Models
• Gated DeltaNet
• In-Context Learning
• Memory Architectures

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence — Structured recurrent states can be parameter-efficient for language modeling but require explicit sparse retrieval for exact long-range recall.
• MemDLM: Memory-Enhanced DLM Training — MemDLM uses bi-level optimization and parametric memory to align DLM training with multi-step inference, improving convergence and long-context understanding.
• Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling — SMET stabilizes dynamic sparse training for LLMs by addressing cold-start issues and optimizing memory, making sparse pre-training practical.
• Memory Caching: RNNs with Growing Memory — Memory Caching enhances RNNs by enabling memory growth with sequence length, bridging the performance gap with Transformers.
• When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet — Multiplication-only matrix inversion approximation significantly speeds up linear attention on NPUs by eliminating sequential dependencies.
• Introduction to Recurrent Neural Networks ,LSTM, GRU’s in Deep learning — Recurrent Neural Networks and their variants process sequential data by maintaining internal memory to capture context and order.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.