From SGD to Muon: Adaptive Optimization via Schatten-p Norms

2026-05-19 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A novel adaptive optimizer, Muon, introduces a data-driven criterion for dynamically selecting proxy-optimal Linear Minimization Oracle (LMO) geometries for individual Deep Neural Network layers. This method, derived in closed form from gradient and activation statistics using a single-step random feature regression surrogate model, navigates a design space that interpolates between SGD and Muon updates. The framework, by integrating parameter-wise preconditioning, can recover SGD, Muon, Adam, and MuAdam as specific extrema. This adaptive approach achieves scalability with only a ~3% runtime overhead on highly optimized baselines. Proof-of-concept results demonstrate that this data-driven optimizer beats or remains competitive with the best performance between Muon and AdamW across three distinct training scenarios.

Key takeaway

For Machine Learning Engineers designing or selecting optimizers, this work suggests that dynamically adapting LMO geometry offers a promising pathway beyond static update rules. You should consider exploring data-driven adaptive optimization techniques, like the proposed Muon variant, to potentially achieve superior or competitive performance with minimal runtime overhead compared to established methods like AdamW.

Key insights

Dynamically adapting LMO geometry from runtime data improves optimizer performance and design.

Principles

• LMO geometry can be successfully adapted from runtime data.
• Adaptive optimizers can unify diverse update rules.

Method

A data-driven criterion, derived from gradient and activation statistics via a single-step random feature regression surrogate model, dynamically chooses LMO geometries.

In practice

• Achieves ~3% runtime overhead on optimized baselines.
• Recovers SGD, Muon, Adam, and MuAdam as specific extrema.

Topics

• Adaptive Optimization
• Schatten-p Norms
• LMO Theory
• Deep Neural Networks
• SGD
• Muon
• AdamW

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Schattor: Schatten-family methods for deep learning optimization — Schattor unifies deep learning optimizers like SGD and Muon using Schatten norms, offering dimension-free stationarity guarantees for complex optimization problems.
• LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold — LoRA-Muon applies spectral steepest descent to LoRA, improving finetuning stability, learning rate transferability, and performance over dense baselines.
• Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning — AdaNAGED unifies gradient-free, adaptive, and geometry-aware optimization for memory-efficient LLM fine-tuning.
• Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks — A curvature-aware optimization framework improves PINN training by adaptively correcting first-order optimizers using secant information.
• First-Principles Optimizer Matches Adam on CIFAR…No Tuning — A first-principles optimizer dynamically adapts learning rates, matching Adam's performance without hyper-parameter tuning.
• Stochastic convergence of parallel asynchronous adaptive first-order methods — Asynchronous adaptive first-order methods achieve O(1/sqrt{t}) convergence on non-convex functions in stochastic settings.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.