Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps

2026-07-14 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A recent study provides a provable explanation for why the cooldown phase in warmup-stable-decay (WSD) learning-rate schedules, a default in large-model pretraining, sometimes reduces final training loss and sometimes has no effect. The research identifies two joint determinants: the structure of gradient noise and whether the optimizer normalizes its update. For strongly convex objectives with multiplicative (gradient-proportional) noise, stochastic gradient descent (SGD) contracts geometrically at a constant learning rate, rendering cooldown unnecessary. In contrast, sign-based and normalized methods, which model adaptive optimizers, establish an $η^2$ noise floor, only reaching the minimizer as the learning rate approaches zero; additive noise then creates a floor for all methods. This occurs because SGD steps shrink proportionally to the gradient, self-annealing, while normalized steps maintain unit scale. The study exactly solves the signSGD stationary law on a quadratic, extends the noise floor to normalized SGD in dimensions d>1 via scale-invariance, and confirms robustness to momentum and heavy-tailed noise through simulation.

Key takeaway

For Machine Learning Engineers designing learning-rate schedules, understand that cooldown phase efficacy hinges on your optimizer's normalization and the gradient noise structure. If you use normalized optimizers or encounter additive noise, a cooldown phase is crucial. It helps drive the learning rate to zero, overcoming the $η^2$ noise floor. Conversely, with SGD and multiplicative noise, cooldown offers minimal benefit. Evaluate your specific training setup to optimize schedule design.

Key insights

Gradient noise structure and optimizer normalization jointly determine learning-rate cooldown efficacy.

Principles

• SGD steps shrink with gradient magnitude.
• Normalized optimizers maintain unit step scale.
• Multiplicative noise allows SGD geometric contraction.

Method

The paper solves the signSGD stationary law on a quadratic, extends the noise floor to normalized SGD in d>1 using scale-invariance, and establishes robustness to momentum and heavy-tailed noise.

In practice

• Diagnose noise regime on classification tasks.
• Evaluate optimizer choice against noise structure.
• Adjust learning rate schedules based on optimizer type.

Topics

• Learning Rate Schedules
• Optimizer Normalization
• Gradient Noise
• Stochastic Gradient Descent
• Adaptive Optimizers
• WSD Cooldown

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Why Gradient Descent Is Just Physics in Disguise — Neural network training is a physical simulation governed by energy, friction, and temperature.
• Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay — Weight-norm criticality, induced by normalization and weight decay, causes loss spikes by driving scale-invariant weight norms to zero.
• Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures — Learning rate scheduler performance is architecture-dependent, with advanced strategies like CosineAnnealingWarmRestarts often outperforming basic decay.
• Learning to be uncertain before learning from data — Training neural networks on random noise before data exposure improves calibration and reliability.
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation — Extensive-width shallow neural networks exhibit a phase transition between universal and specialization learning, impacting generalization and algorithmic tractability.
• Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization — Hyperball fixes weight matrix Frobenius norms to improve matrix optimizer performance in large model pretraining.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.