Gated DeltaNet-2: Better Memory Editing for Linear Attention

2026-04-15 · Source: The Kaitchup – AI on a Budget · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

The latest intelligence brief highlights three significant developments in large language models. Gated DeltaNet-2 (GDN-2) refines linear attention by decoupling memory erase and write operations into two channel-wise gates, improving upon the original GDN used in Qwen3.5-397B-A17B and Qwen3.6-27B. While theoretically sound, its empirical validation is questioned due to outdated, logit-based benchmarks. NVIDIA introduced Nemotron-Labs-Diffusion, a hybrid autoregressive/diffusion model family (3B, 8B, 14B) capable of normal AR, parallel diffusion, and self-speculation decoding. This architecture achieves up to 6 tokens per forward pass in self-speculation mode, offering substantial speed gains at low concurrency while maintaining AR-level quality. Finally, Cohere released Command A+, an Apache 2.0 licensed decoder-only Transformer with 218B total parameters and 25B active per token, featuring a dropless Mixture-of-Experts architecture and parallel MoE/attention computation. It is available in BF16, FP8, and W4A4 formats, scoring 37 on the Artificial Analysis Intelligence Index, positioning it competitively.

Key takeaway

For machine learning engineers evaluating new LLM architectures, consider the trade-offs between architectural precision and empirical validation. Gated DeltaNet-2 offers refined memory editing, but its real-world performance needs modern benchmarks. If you prioritize inference speed, Nemotron-Labs-Diffusion provides significant gains at low concurrency through hybrid decoding. For commercial applications requiring an open-source MoE model, Command A+ is a competitive Apache 2.0 option, but verify its performance on your specific workloads.

Key insights

The latest LLM advancements focus on architectural refinements and hybrid decoding for improved efficiency and performance.

Principles

• Decoupling memory operations enhances linear attention.
• Hybrid AR/diffusion models boost decoding speed.
• MoE architectures can be optimized for deployment.

Method

Gated DeltaNet-2 refines linear attention by splitting the scalar update gate into distinct channel-wise gates for key-side erase and value-side write, enabling precise memory editing. Nemotron-Labs-Diffusion uses a decoder-only Transformer with special attention masks and joint training to enable AR, diffusion, and self-speculation modes.

In practice

• Use GDN-2 for refined linear attention memory.
• Deploy Nemotron-Labs-Diffusion for faster token generation.
• Consider Command A+ for Apache 2.0 MoE models.

Topics

• Gated DeltaNet-2
• Linear Attention
• Hybrid LLMs
• Mixture-of-Experts
• Model Quantization
• LLM Benchmarking

Code references

• NVlabs/GatedDeltaNet-2

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention — Decoupling erase and write operations with channel-wise gates significantly enhances linear attention's memory editing capabilities.
• NVIDIA AI Releases Nemotron-Labs-Diffusion: A Tri-Mode Language Model with 6× Tokens Per Forward Over Qwen3-8B — Nemotron-Labs-Diffusion unifies AR and diffusion for efficient, accurate LLM inference via self-speculation.
• A Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan-Feb 2026 — Recent open-weight LLMs prioritize Mixture-of-Experts and hybrid attention for efficiency and performance across diverse scales.
• Qwen3.5: Scaling Hybrid Attention to 397B Parameters — Hybrid attention with Gated DeltaNet scales effectively, enabling cost-efficient, high-performing LLMs.
• The Evolution of LLM Inference: Decoding algorithms — Part 1 — LLM inference speed hinges on reducing sequential decoding steps by predicting and verifying multiple tokens in parallel.
• DOT-MoE: Differentiable Optimal Transport for MoEfication — DOT-MoE converts dense LLMs to sparse MoEs by framing neuron decomposition as a Differentiable Optimal Transport problem for efficient inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Kaitchup – AI on a Budget.