The Spectral Dynamics and Noise Geometry of Muon

2026-06-07 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Muon is an optimization method that replaces a matrix gradient G=UΣV^⁴ with its polar factor UV^⁴, preserving singular directions while flattening the update spectrum. This study investigates the optimization bias created by Muon. Under explicit alignment assumptions, it proves the polar update is the one-step entropy-maximizing choice among bounded updates using gradient singular directions. For an underdetermined regression model, exact singular-value dynamics for continuous-time Muon are derived, identifying a measurement-dependent condition for the normalized spectrum to move toward equal nonzero singular values. This geometry contradicts a low-rank interpretation, as Muon favors a flat spectrum at fixed Frobenius norm, unlike nuclear-norm minimization. Experiments confirm Muon's unique behavior and the flattening trend. In small NanoGPT pretraining, Muon preserves stable rank, offers a broad learning-rate plateau, and improves validation loss over AdamW, but AdamW performs better in a small-ViT control. Muon's flat-spectrum bias is thus regime-dependent, beneficial when many spectral directions need to remain active.

Key takeaway

For Machine Learning Engineers optimizing deep learning models, you should consider Muon when your architecture benefits from maintaining activity across many spectral directions, such as in NanoGPT pretraining where it improved validation loss. However, be aware that its performance is regime-dependent; for models like small-ViT, AdamW may still be superior. Evaluate Muon's flat-spectrum bias against your specific model's needs and conduct comparative ablations.

Key insights

Muon's polar factor gradient update flattens the spectrum, offering regime-dependent benefits by keeping many spectral directions active.

Principles

• Polar updates can be entropy-maximizing for bounded updates.
• Flat spectrum bias distinguishes Muon from nuclear-norm minimization.
• Optimization method benefits are often regime-dependent.

Method

Muon replaces the matrix gradient G=UΣV^⁴ with its polar factor UV^⁴, effectively flattening the update spectrum while retaining singular directions.

In practice

• Consider Muon for models needing active spectral directions.
• Evaluate Muon against AdamW for specific model architectures.
• Note Muon's broad learning-rate plateau in NanoGPT.

Topics

• Muon Optimizer
• Spectral Dynamics
• Gradient Descent
• Deep Learning Optimization
• NanoGPT Pretraining
• Vision Transformers

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy — LLM gradients exhibit a "spike-tail" anisotropy where dominant linguistic structures suppress long-tail semantic learning.
• De novo molecular generation with optical property preconditioning at the token level — Token-conditioned autoregressive models can generate OLED molecules with targeted optical properties, but control varies by chemical motif.
• Muon Learns More Robust and Transferable Features than Adam — Muon optimizer yields more robust and transferable features than Adam and SGD across diverse model architectures and tasks.
• From SGD to Muon: Adaptive Optimization via Schatten-p Norms — Dynamically adapting LMO geometry from runtime data improves optimizer performance and design.
• Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network — A new model tracks average trajectory and oscillation covariance, revealing an effective free energy for Edge of Stability dynamics.
• Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate — Maximizing the embedding layer learning rate is crucial for effective hyperparameter transfer in large language model training.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.