Training Diagonal Linear Networks with Stochastic Sharpness-Aware Minimization

2026-07-14 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

A theoretical analysis of Stochastic Sharpness-Aware Minimization (S-SAM) in diagonal linear networks reveals its regularization effects. S-SAM introduces isotropic normal noise (parameter η) during training, which acts as a regularizer, R, equivalent to minimizing average sharpness. For diagonal linear networks, S-SAM forces weight matrices to be "balanced" (e.g., Wℓ^2 = Wm^2), which in turn minimizes average sharpness and the trace of the Hessian matrix. The method drives parameters towards a shrinkage-thresholding of the true underlying parameter W*, with η and network depth L controlling these effects. This regularization ensures convergence to critical points of the regularized loss, both in continuous gradient flow and discrete gradient descent, and is shown to improve generalization performance.

Key takeaway

For AI Scientists and Machine Learning Engineers optimizing deep linear models for better generalization, S-SAM offers a theoretically grounded approach. You should implement S-SAM to explicitly control loss landscape sharpness and parameter norms. Carefully tune the noise parameter η to achieve balanced weight matrices and desired shrinkage, especially for models prone to sharp minima or large parameter norms, thereby enhancing model robustness and generalization.

Key insights

S-SAM regularizes diagonal linear networks by balancing weights, minimizing sharpness, and inducing parameter shrinkage for better generalization.

Principles

• Algorithmic noise can induce explicit regularization effects on loss landscapes.
• Balancing conditions in deep linear networks minimize average sharpness and Hessian trace.
• Sharpness-aware minimization can lead to shrinkage-thresholding of true parameters.

Method

S-SAM trains diagonal linear networks by perturbing network parameters with small isotropic normal noise N(0, η^2) during gradient descent iterations.

In practice

• Consider S-SAM for deep linear models to achieve balanced weight matrices.
• Adjust the noise level η to control parameter shrinkage and thresholding effects.

Topics

• Stochastic Sharpness-Aware Minimization
• Diagonal Linear Networks
• Loss Landscape Optimization
• Model Generalization
• Algorithmic Regularization
• Gradient Descent Dynamics
• Weight Balancing

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Optimizing Deep Learning Models with SAM — Sharpness-Aware Minimization (SAM) improves deep learning model generalizability by addressing overparameterization challenges.
• From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning — Sharpness-Guided Equilibrium Sampling (SGS) dynamically adjusts class sampling to achieve balanced flat minima in long-tailed learning.
• A Function-Centric Perspective on Flat and Sharp Minima — Sharpness in neural network minima is function-dependent, not solely indicative of poor generalization.
• Adaptive deep nonparametric regression from dependent data under covariate shift — SPDNN estimators achieve minimax optimal rates for nonparametric regression under covariate shift with dependent data.
• Why Not Hyperparameter-Friendly Optimisation? A Monotonic Adaptive Norm Rescaling Approach For Long-Tailed Recognition — SAMN improves long-tailed recognition by enforcing monotonic per-class weight norms without sensitive hyperparameters.
• Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization — S-Adam stabilizes non-smooth deep learning optimization by dynamically adjusting step sizes based on local geometric instability, preventing gradient chattering.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.