Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention

2026-07-10 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new learnable Dirichlet-process cache is introduced to address the memory and recall limitations of existing sequence models. This sparse cache allocates memory slots only for novel inputs, tracking distinct items rather than every token, unlike attention mechanisms which incur quadratic compute and growing cache sizes, or fixed-state models with capped associative recall. Developed in static and surprise-adaptive forms, the cache utilizes a DP-means clustering rule as a key-value memory operator. Benchmarks show it matches full-attention recall while storing only distinct items, outperforms fixed-budget eviction caches, and achieves the lowest memory for recall and long-range aggregate queries on a state-space backbone. Its allocation is end-to-end learnable, with a two-parameter novelty-threshold gate recovering the rule. The distinct-items property was confirmed on recommendation, systems logs, clinical events, and insurance claims data streams.

Key takeaway

For Machine Learning Engineers optimizing sequence models for long-range dependencies and memory efficiency, you should investigate implementing a distinct-item sparse cache. This approach allows your models to achieve full-attention recall performance while drastically reducing memory footprint by storing only novel inputs. Focus on designing learnable allocation rules with strong inductive biases, as over-parameterized gates may fail to recover the optimal mechanism.

Key insights

A sparse cache tracking distinct items, not tokens, significantly improves sequence model recall efficiency.

Principles

• Sparse caches can achieve full-attention recall with reduced memory.
• Novelty-driven memory allocation enhances efficiency for redundant sequences.
• Inductive bias is critical for effective end-to-end learnable allocation rules.

Method

The proposed method uses a DP-means clustering rule as a key-value memory operator for a deep recurrent backbone, allocating slots only for novel inputs. It comes in static and surprise-adaptive variants.

In practice

• Implement a distinct-item sparse cache in sequence models.
• Apply DP-means clustering for novelty-based memory allocation.
• Train a two-parameter novelty-threshold gate for allocation.

Topics

• State-Space Models
• Attention Mechanisms
• Sparse Caches
• Dirichlet Process
• Associative Recall
• Memory Efficiency

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence — Structured recurrent states can be parameter-efficient for language modeling but require explicit sparse retrieval for exact long-range recall.
• Memory Caching: RNNs with Growing Memory — Memory Caching enhances RNNs by enabling memory growth with sequence length, bridging the performance gap with Transformers.
• EntmaxKV: Support-Aware Decoding for Entmax Attention — EntmaxKV enables efficient, exact sparse decoding for long-context models by exploiting Entmax's zero-producing property.
• Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity — Sparse Delta Memory scales linear RNNs' state for superior long-context and reasoning performance.
• SR-TTT Does Not Learn Retrieval: A Correction and Mechanistic Post-Mortem of Surprisal-Aware Residual Test-Time Training — SR-TTT resolves TTT's exact-recall failures by dynamically routing incompressible tokens to a sparse, exact-attention Residual Cache.
• Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams — A novel working memory scales context by distinct information, not tokens, improving efficiency and auditability.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.