This Simple Optimizer Is Revolutionizing How We Train AI [Muon]

2025-10-14 · Source: Jia-Bin Huang · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Advanced, long

Summary

The Muon Optimizer is presented as an efficient alternative to AdamW for training machine learning models, particularly small language models. It achieves approximately twice the computational efficiency of AdamW by addressing the limitations of vector-based optimizers, which treat parameters as a single long vector. Muon introduces an orthogonalization process for momentum matrices, specifically for linear layers, to amplify the effect of rare directions in parameter updates. This orthogonalization is efficiently performed using an odd polynomial matrix function, which pushes singular values closer to one without computationally intensive Singular Value Decomposition (SVD). The algorithm involves computing gradients, updating momentum, normalizing the 2D momentum matrix, and repeating the orthogonalization process. For larger models, Muon incorporates weight decay and adjusts the learning rate based on matrix size. Additionally, the QK clip and Muon clip mechanisms are introduced to stabilize training by preventing attention logits from growing excessively large, especially in multi-head attention and Multi-Head Latent Attention (MLA) architectures, by selectively rescaling query and key projection weights.

Key takeaway

For NLP Engineers and AI Scientists training language models, consider adopting the Muon Optimizer. Its orthogonalization technique and attention logit clipping mechanisms (QK clip, Muon clip) offer significant computational efficiency and improved training stability, particularly for small language models and large models with multi-head attention. Your models could train faster and more reliably, reducing resource consumption and mitigating instability issues commonly seen with AdamW.

Key insights

Muon Optimizer enhances training efficiency and stability by orthogonalizing momentum and clipping attention logits.

Principles

• Orthogonalization amplifies rare update directions.
• Odd polynomial matrices efficiently approximate SVD.
• Attention logit clipping stabilizes large model training.

Method

Muon computes gradients, updates momentum, normalizes 2D momentum matrices, and applies an odd polynomial matrix function five times for orthogonalization, then updates parameters. QK clip and Muon clip rescale query/key weights to control attention logit scale.

In practice

• Use Muon for faster, memory-efficient training.
• Apply QK clip to stabilize attention logits.
• Implement Muon clip for multi-head latent attention.

Topics

• Muon Optimizer
• Adam Optimizer
• Orthogonalization
• Attention Mechanisms
• QK Clip

Best for: NLP Engineer, AI Scientist, Research Scientist, AI Engineer, Machine Learning Engineer, AI Researcher

Related on AIssential

• The First Optimizer to Challenge Adam in a Decade Just Cut Training Costs in Half — Muon optimizer offers significant compute efficiency gains, challenging Adam's long-standing dominance in deep learning.
• [R] AdamWClip: AdamW with adaptive gradient clipping — AdamWClip is an AdamW extension offering adaptive gradient clipping without manual thresholds or significant overhead.
• Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models — Octopus enables continual learning in MLLMs by orthogonalizing gradients without needing historical data.
• DMuon: Efficient Distributed Muon Training with Near-Adam Overhead — DMuon efficiently scales matrix-orthogonalization optimizers for distributed deep learning, achieving near-AdamW performance.
• ​Sequential Attention: Making AI models leaner and faster without sacrificing accuracy — Sequential Attention makes large ML models leaner and faster by sequentially selecting optimal features and components.
• Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate — Maximizing the embedding layer learning rate is crucial for effective hyperparameter transfer in large language model training.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Jia-Bin Huang.