A Defense of the Quadratic Model

2026-07-23 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study on "A Defense of the Quadratic Model" demonstrates that the simple quadratic model can surprisingly predict optimization dynamics in Large Language Model (LLM) training. Researchers stress-tested this model using an LLM with 150M parameters and 3B training tokens, finding it accurately predicts dynamics over windows up to 10% of total training. By Taylor expanding the model and loss function at intermediate checkpoints, they established this agreement. Further analysis explored the structure of local quadratic optimization problems through the Hessian spectrum and local stability. The study revealed significant structure in both eigenvalues and eigenvectors, influenced by batch size, preconditioner, and training time. Empirical tests confirmed LLM optimization typically occurs at a stochastic edge of stability, also determined by batch size, suggesting the quadratic model is a tractable proxy for pretraining optimization.

Key takeaway

For research scientists investigating LLM optimization dynamics, you should consider the quadratic model as a powerful, tractable proxy. This model can predict training behavior over significant windows, offering insights into the Hessian spectrum and local stability. Use these findings to better understand how batch size and preconditioners influence your model's optimization path and stability, potentially guiding hyperparameter tuning.

Key insights

The quadratic model offers a surprisingly predictive and theoretically tractable proxy for understanding LLM pretraining optimization dynamics.

Principles

• Quadratic models can predict LLM optimization dynamics.
• Hessian spectrum and stability reveal optimization structure.
• Batch size influences optimization stability and Hessian structure.

Method

Taylor expand the quadratic model and loss function at intermediate training checkpoints. Estimate Hessian spectrum using Lanczos quadrature with deep probes. Empirically test local linear stability.

In practice

• Analyze Hessian spectrum to understand training.
• Monitor local stability for optimization insights.
• Adjust batch size to influence stability.

Topics

• Quadratic Model
• LLM Optimization
• Neural Network Training
• Hessian Spectrum
• Local Stability
• Batch Size

Best for: AI Scientist, Research Scientist

Related on AIssential

• A Defense of the Quadratic Model — The quadratic model accurately predicts LLM optimization dynamics locally, offering a tractable proxy for pretraining analysis.
• PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training — Polynomial preconditioning reshapes weight singular-value spectra to improve LLM training stability and efficiency without inference overhead.
• On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials — Nonlinear dynamics significantly impact GD/SGD stability, often contradicting linear approximations, especially with high-order derivatives and single batch instability.
• Zeroth-Order Optimization at the Edge of Stability — Zeroth-order optimization stability depends on the full Hessian spectrum, unlike first-order methods.
• The Impact of Dimensionality on the Stability of Node Embeddings — Node embedding stability varies with dimensionality, but optimal stability does not always align with peak performance.
• A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Prediction — The framework decomposes pre-hoc fine-tuning prediction risk into intrinsic and reducible components, revealing fundamental uncertainty limits.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.