Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training

2026-03-18 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

MUD (MomentUm Decorrelation) is a new optimizer designed to accelerate Transformer training, offering 10-50% wall-clock time improvements over tuned AdamW and Muon in time-to-perplexity. MUD replaces Muon's polar decomposition-based orthogonalized-momentum updates with a triangular (Cholesky-like) whitening surrogate, drawing inspiration from Gram--Schmidt and Gauss-Seidel methods. This approach significantly reduces optimizer overhead, improving peak tokens/s by approximately 1.3-2.6x across most settings, and up to nearly 3x on GPT-2 large models running on A100 GPUs, compared to Muon. While MUD converges slightly slower per step than Muon, its lower overhead leads to faster overall training times, as demonstrated by matching Muon-level validation perplexity for an ESM-2 150M protein language model in less wall-clock time.

Key takeaway

For AI Engineers optimizing large language model training, MUD offers a significant opportunity to reduce wall-clock training time. If your current setup uses AdamW or Muon, consider integrating MUD to achieve 10-50% faster convergence to target perplexity, especially on A100 GPUs. This could translate to substantial cost savings and quicker iteration cycles for model development and deployment.

Key insights

MUD optimizer accelerates Transformer training by replacing polar decomposition with a Cholesky-like whitening surrogate, reducing overhead.

Principles

• Row-orthonormal matrices are MUD map fixed points.
• Inner step relates to symmetric Gauss-Seidel preconditioning.

Method

MUD uses a triangular (Cholesky-like) whitening surrogate, inspired by Gram--Schmidt and Gauss-Seidel, to decorrelate momentum updates, replacing Muon's polar decomposition.

In practice

• Achieves 10-50% wall-clock speedup.
• Improves peak tokens/s by 1.3-2.6x.
• Trains ESM-2 150M faster.

Topics

• Transformer Training
• Optimization Algorithms
• Momentum Optimizers
• Whitening Techniques
• Protein Language Models

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• The First Optimizer to Challenge Adam in a Decade Just Cut Training Costs in Half — Muon optimizer offers significant compute efficiency gains, challenging Adam's long-standing dominance in deep learning.
• Surrogate models for Rock-Fluid Interaction: A Grid-Size-Invariant Approach — Grid-size-invariant neural networks offer efficient, scalable surrogate modeling for complex rock-fluid interactions.
• How to Optimize Transformer-Based Models for Low-Precision Training — Optimizing low-precision Transformer training requires benchmarking actual GEMM workloads to understand real speedups.
• Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency — Controlled weight inconsistency in asynchronous pipeline training boosts efficiency without sacrificing stability.
• Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy — LLM gradients exhibit a "spike-tail" anisotropy where dominant linguistic structures suppress long-tail semantic learning.
• Hierarchical Transformer Preconditioning for Interactive Physics Simulation — A hierarchical transformer preconditioner improves real-time physics simulation by efficiently capturing long-range couplings with O(N) scaling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.