Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics

2026-06-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

This theoretical study investigates feed-forward ReLU networks utilizing fixed readout and quadratic loss, reframing gradient descent as a collective dynamics within the training-set space rather than the traditional weight space. For networks with a single hidden layer, the weight variables are eliminated from the activation dynamics, resulting in a closed equation for the residuals. This equation is governed by a collective kernel that factorizes into an input-geometric matrix and a dynamical co-activation matrix. As network depth increases, the residual dynamics maintains a clean layer-wise kernel structure. However, the research reveals that for networks from depth three onward, achieving closure necessitates a hierarchy of weight-induced Gram operators, which are crucial for mediating information transport across the network's layers.

Key takeaway

For AI Scientists researching neural network optimization, this work suggests a fundamental shift in understanding gradient descent. You should consider modeling learning dynamics not just in weight space, but as collective dynamics within the training-set space, especially when analyzing deeper ReLU networks. This perspective highlights the critical role of layer-wise Gram operators in information transport, potentially guiding new approaches to network design and training stability.

Key insights

Gradient descent in ReLU networks can be modeled as collective dynamics in training-set space, mediated by layer-wise Gram operators.

Principles

• Gradient descent can be reframed as collective dynamics.
• Deeper networks require a hierarchy of Gram operators.
• Weight variables can be eliminated for single-layer networks.

Method

The study proposes rewriting gradient descent for feed-forward ReLU networks as a collective dynamics closed in terms of fields defined on the training-set space, eliminating weight variables for single-layer cases.

Topics

• ReLU Networks
• Gradient Descent
• Learning Dynamics
• Gram Operators
• Feed-forward Networks
• Neural Network Optimization

Best for: Research Scientist, AI Scientist

Related on AIssential

• Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network — Effective free energy, combining risk and curvature, is key to understanding gradient descent dynamics at the Edge of Stability.
• Foundations of Deep Learning: Gradient Descent, Back Propagation and Self-Attention — Gradient Descent, Back Propagation, and Self-Attention are core algorithms enabling modern neural networks and LLMs.
• How does feature learning reshape the function space? — Feature learning in two-layer neural networks deforms function space via a data-adaptive, target-dependent kernel.
• Classification Fields: Arbitrarily Fine Recursive Hierarchical Clustering From Few Examples — Classification fields enable learning infinite-depth hierarchical cluster structures from finite observations via local refinement rules.
• Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime — A universal theoretical framework for neural scaling laws and weight spectra in shallow networks is established via sparse estimation.
• Layerwise Dynamics for In-Context Classification in Transformers — Enforcing task symmetries reveals a closed-form, layerwise algorithmic recursion in transformers for in-context classification.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.