Tensorion: A Tensor-Aware Generalization of the Muon Optimizer

2026-06-24 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, medium

Summary

Tensorion is a novel tensor-aware optimizer that generalizes the Muon optimizer, extending its constrained optimization approach from matrices to higher-order tensors. Unlike common first-order optimizers like Adam, which treat parameters as unstructured vectors, Tensorion explicitly accounts for the multilinear weight structure prevalent in modern machine learning models. It operates by performing linear minimization over a carefully chosen tensor norm ball, balancing a tight bound on the tensor spectral norm with LMO tractability. This LMO is computable by reducing operations to adaptively selected unfolding matrices. When applied to order-2 tensors, Tensorion precisely recovers Muon. Experimental evaluations on tensor-based computer vision problems indicate that Tensorion offers improved convergence behavior and more stable gradient updates compared to Adam-based and existing tensor-aware baselines.

Key takeaway

For Machine Learning Engineers optimizing models with higher-order tensor weight structures, you should evaluate Tensorion as an alternative to Adam-based optimizers. Its tensor-aware approach, generalizing Muon, can provide improved convergence and more stable gradient updates, particularly in computer vision tasks. Consider integrating Tensorion to potentially enhance training efficiency and model performance in your tensor-centric applications.

Key insights

Tensorion extends matrix-aware optimization to higher-order tensors via a tractable linear minimization oracle over a tensor norm ball.

Principles

• Exploiting multilinear weight structure improves optimization dynamics.
• Tensor-aware optimizers can offer more stable gradient updates.

Method

Tensorion's LMO is computed by reducing operations to adaptively selected unfolding matrices, ensuring tractability while tightly bounding the tensor spectral norm.

In practice

• Apply Tensorion to tensor-based computer vision problems.
• Consider Tensorion for models with higher-order tensor weight structures.

Topics

• Tensorion
• Muon Optimizer
• Tensor Networks
• Optimization Algorithms
• Computer Vision
• Machine Learning Models

Code references

• Saurabh750/optimizer
• K1seki221/MuonPlus

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• The Spectral Dynamics and Noise Geometry of Muon — Muon's polar factor gradient update flattens the spectrum, offering regime-dependent benefits by keeping many spectral directions active.
• EinSort: Sorting is All We Need for Tensorizing LLM — EinSort uses index ordering for adaptive tensorization to compress LLMs by finding low-rank structures.
• Optimizers in Deep Learning: From Gradient Descent to Adam — Optimizers guide neural network training by efficiently updating parameters to minimize loss, evolving from basic GD to adaptive Adam.
• A Typed Tensor Language for Federated Learning — The language formalizes federated learning computations, enabling fixed-dimensional shared state communication for efficiency.
• Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks — A curvature-aware optimization framework improves PINN training by adaptively correcting first-order optimizers using secant information.
• From SGD to Muon: Adaptive Optimization via Schatten-p Norms — Dynamically adapting LMO geometry from runtime data improves optimizer performance and design.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.