Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new mechanism called "weight-norm criticality" has been identified as a cause for loss spikes during deep neural network training, distinct from the well-known "learning-rate criticality." This phenomenon arises from the interaction between normalization techniques, which introduce scale-invariant components, and weight decay, which consistently reduces parameter norms. As the weight decay coefficient increases, the norms of these scale-invariant weights are progressively driven towards zero. This process simultaneously increases the sharpness of the loss landscape, destabilizing optimization dynamics and leading to abrupt loss spikes. This understanding clarifies why weight penalties enhance generalization but become detrimental if applied too strongly, pushing scale-invariant weight norms past a critical boundary. The proposed mechanism has been empirically validated in networks featuring scale-invariant components.

Key takeaway

For Machine Learning Engineers tuning hyperparameters in deep neural networks, you should recognize that weight decay is not arbitrarily scalable for generalization benefits. Your optimization stability is critically dependent on avoiding "weight-norm criticality," where excessive weight decay drives scale-invariant weight norms too low. Monitor loss spikes in relation to your weight decay coefficient, especially in models using normalization, to prevent destabilizing training dynamics and ensure robust model performance.

Key insights

Weight-norm criticality, induced by normalization and weight decay, causes loss spikes by driving scale-invariant weight norms to zero.

Principles

• Normalization introduces scale-invariant components.
• Weight decay persistently shrinks parameter norms.
• Excessive weight decay destabilizes optimization.

Topics

• Weight-norm Criticality
• Loss Spikes
• Deep Neural Networks
• Normalization
• Weight Decay
• Optimization Stability

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Deep Neural Regression Collapse — Deep Neural Regression Collapse extends beyond the last layer, revealing structured feature learning in deep networks.
• What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy — Weight norm controls grokking delay primarily through logit scale and softmax saturation, especially under cross-entropy.
• Neural Network HyperParameters — Hyperparameters define neural network structure and training, requiring careful tuning for optimal performance and generalization.
• Adversarial Rademacher Complexity of Deep Neural Networks — This paper establishes the first theoretical bound for Adversarial Rademacher Complexity in deep neural networks, linking robust generalization to weight norm.
• Why Every Weight in a Neural Network Is Born Divided by the Square Root of n. — Proper neural network weight initialization prevents exploding or vanishing signals, ensuring stable learning from the start.
• Preserving Plasticity in Continual Learning via Dynamical Isometry — Dynamical isometry is a key mechanism for preserving plasticity in continual learning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.