NVIDIA AI Releases Gated DeltaNet-2: A Linear Attention Layer That Decouples Erase and Write in the Delta Rule

2026-05-24 · Source: Machine Learning ML & Generative AI News · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

NVIDIA AI has released Gated DeltaNet-2, a new linear attention layer designed to improve memory management in large language models. This model addresses a key limitation in prior delta-rule architectures like Gated DeltaNet and KDA, which used a single scalar gate to simultaneously erase old key-side content and write new value-side content. Gated DeltaNet-2 introduces separate channel-wise gates: an erase gate (b_t) for removing key-side coordinates and a write gate (w_t) for committing new value-side content. This decoupling allows for more precise control over memory editing. The architecture also offers strict generalization, recovering previous models under specific conditions, and maintains fast training speeds through a Chunkwise WY algorithm and Triton-fused backward passes. Benchmarked at 1.3B parameters on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieved superior language modeling and commonsense averages, notably improving S-NIAH-3 scores from KDA's 63.2 to 89.8 and MK-NIAH-1 scores from 28.0 to 37.8.

Key takeaway

For Machine Learning Engineers optimizing recurrent state memory in large language models, NVIDIA's Gated DeltaNet-2 presents a significant architectural advancement. Its decoupled channel-wise erase and write gates offer superior control over memory editing compared to previous delta-rule models. You should explore its open-source implementation to enhance language modeling and commonsense reasoning, especially when working with fixed-size recurrent states. This could lead to better performance and more efficient resource utilization in your projects.

Key insights

Gated DeltaNet-2 improves linear attention by decoupling memory erase and write operations with channel-wise gates.

Principles

• Decoupling memory operations enhances control.
• Channel-wise gating offers granular state editing.
• Generalization allows backward compatibility.

Method

Gated DeltaNet-2 uses channel-wise erase (b_t) and write (w_t) gates. Training employs a Chunkwise WY algorithm with gate-aware backward fused in Triton.

In practice

• Implement channel-wise gates for memory control.
• Use Chunkwise WY for efficient training.
• Benchmark against Mamba-2, Mamba-3 for recurrent state.

Topics

• Linear Attention
• Gated DeltaNet-2
• Recurrent State Memory
• Large Language Models
• NVIDIA AI
• Delta Rule Architectures

Code references

• NVlabs/GatedDeltaNet-2

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention — Decoupling erase and write operations with channel-wise gates significantly enhances linear attention's memory editing capabilities.
• Attention from First Principles: DeltaNet — DeltaNet optimizes linear attention by correcting errors rather than redundantly rewriting known information.
• DualOptim+: Bridging Shared and Decoupled Optimizer States for Better Machine Unlearning in Large Language Models — DualOptim+ adaptively balances shared and decoupled optimizer states for superior LLM unlearning.
• EntmaxKV: Support-Aware Decoding for Entmax Attention — EntmaxKV enables efficient, exact sparse decoding for long-context models by exploiting Entmax's zero-producing property.
• Two-dimensional early exit optimisation of LLM inference — Coordinating layer-wise and sentence-wise early exits multiplicatively boosts LLM inference speed for classification.
• Gated DeltaNet-2: Better Memory Editing for Linear Attention — The latest LLM advancements focus on architectural refinements and hybrid decoding for improved efficiency and performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning ML & Generative AI News.