Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

2026-05-21 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Gated DeltaNet-2 is a novel linear attention model that significantly advances memory editing by decoupling the erase and write functions within its recurrent state. Generalizing both Gated DeltaNet and Kimi Delta Attention (KDA), this model introduces distinct channel-wise erase gate b_t and write gate w_t, addressing the scalar tie limitation of previous approaches. Trained with 1.3B parameters on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval benchmarks. Its performance advantage is particularly notable on long-context RULER needle-in-a-haystack tasks, improving multi-key retrieval and maintaining strength in recurrent and hybrid settings.

Key takeaway

For Machine Learning Engineers evaluating linear attention models for long-context applications, Gated DeltaNet-2 offers a compelling solution. Its innovative decoupling of erase and write gates provides superior memory editing, leading to stronger performance in language modeling, commonsense reasoning, and especially long-context retrieval. You should consider integrating Gated DeltaNet-2 to enhance the efficiency and accuracy of your models on complex, memory-intensive tasks.

Key insights

Decoupling erase and write operations with channel-wise gates significantly enhances linear attention's memory editing capabilities.

Principles

• Linear attention replaces unbounded softmax cache with fixed-size recurrent state.
• Effective memory editing requires distinct, channel-wise controls for erasing and writing.
• Adaptive forgetting and channel-wise decay improve memory compression.

Method

Gated DeltaNet-2 derives a fast-weight update view, utilizes a chunkwise WY algorithm with channel-wise decay, and employs a gate-aware backward pass for efficient parallel training.

In practice

• Improves performance on long-context RULER needle-in-a-haystack benchmarks.
• Strong in multi-key retrieval, recurrent, and hybrid settings.

Topics

• Linear Attention
• Gated DeltaNet-2
• Memory Networks
• Language Modeling
• Long-Context Retrieval
• Recurrent Neural Networks

Code references

• NVlabs/GatedDeltaNet-2

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Gated DeltaNet-2: Better Memory Editing for Linear Attention — The latest LLM advancements focus on architectural refinements and hybrid decoding for improved efficiency and performance.
• NVIDIA AI Releases Gated DeltaNet-2: A Linear Attention Layer That Decouples Erase and Write in the Delta Rule — Gated DeltaNet-2 improves linear attention by decoupling memory erase and write operations with channel-wise gates.
• Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention — Recurrent memory models benefit from decoupling memory erase and write operations for improved management.
• Beyond Softmax: The Future of Attention Mechanisms — Linear attention offers constant memory complexity by replacing softmax with a state matrix and recurrence relation.
• Q-Delta: Beyond Key-Value Associative State Evolution — Q-Delta integrates query-aware prediction errors into linear attention's state evolution for improved sequence modeling.
• Context Memorization for Efficient Long Context Generation — Attention-state memory externalizes LLM prefixes into a lookup-based memory, improving long-context inference efficiency and accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.