When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet

2026-06-04 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Hardware & Efficiency · Depth: Expert, quick

Summary

A new MatMul-based algorithm addresses the matrix inversion bottleneck in chunk-wise parallel linear attention, critical for long-context modeling on NPUs. This method, detailed in "When Good Enough Is Optimal," employs a truncated Neumann expansion with structural masking and parallel residual correction to eliminate sequential dependencies. It extends to low-bits INT by mitigating dynamic range expansion from repeated matrix power operations. The algorithm adapts approximation order and residual steps to chunk size, minimizing computational cost while preserving accuracy. Experiments on Qwen3.5-family models show up to 5x kernel-level speedup and a 20% reduction in decode-layer overhead, maintaining accuracy under both floating-point and low-precision inference.

Key takeaway

For Machine Learning Engineers optimizing long-context models on NPUs, this multiplication-only matrix inversion approximation offers a critical performance improvement. You should investigate integrating this MatMul-based approach into your linear attention implementations to achieve up to 5x kernel-level speedups and reduce decode-layer overhead by 20%, especially when targeting low-precision inference. This method provides a hardware-friendly solution for scalable linear attention.

Key insights

Multiplication-only matrix inversion approximation significantly speeds up linear attention on NPUs by eliminating sequential dependencies.

Principles

• Neumann-series terms grow rapidly.
• Inverse matrix shows diagonal concentration.
• Adapt approximation order to chunk size.

Method

A truncated Neumann expansion with structural masking and parallel residual correction, extended to low-bits INT by mitigating dynamic range expansion from repeated matrix power operations.

In practice

• Apply to strictly lower-triangular matrices.
• Optimize for chunk-wise linear attention.
• Mitigate dynamic range expansion for INT.

Topics

• Matrix Inversion
• Linear Attention
• Quantization
• NPUs
• Neumann Expansion
• Qwen3.5 Models

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet — Approximating matrix inversion with MatMul-only operations is optimal for hardware efficiency and low-bit quantization in linear attention.
• [D] A mathematical proof from an anonymous Korean forum: The essence of Attention is fundamentally a d^2 problem, not n^2. (PDF included) — An anonymous paper, "The d^2 Pullback Theorem," claims the intrinsic optimization landscape of the Attention mechanism is fundamentally d^2-dimensional, not n^2, arguing that the O(n^2) bottleneck is…
• EntmaxKV: Support-Aware Decoding for Entmax Attention — EntmaxKV enables efficient, exact sparse decoding for long-context models by exploiting Entmax's zero-producing property.
• NVIDIA AI Releases Gated DeltaNet-2: A Linear Attention Layer That Decouples Erase and Write in the Delta Rule — Gated DeltaNet-2 improves linear attention by decoupling memory erase and write operations with channel-wise gates.
• Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory — KLR-trained Hopfield networks achieve quantization robustness via a "sparse function, dense representation" principle.
• Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime — The paper proposes an equivalent Wishart Ansatz and kernel renormalization to predict Bayesian DNN generalization in the proportional limit.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.