Theory of the Frequency Principle for General Deep Neural Networks

2019-06-21 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

The paper "Theory of the Frequency Principle for General Deep Neural Networks" by Tao Luo, Zheng Ma, Zhi-Qin John Xu, and Yaoyu Zhang, submitted on June 21, 2019, and last revised June 25, 2026, rigorously investigates the universal Frequency Principle (F-Principle) in Deep Neural Networks (DNNs). This principle describes how DNNs learn target functions by prioritizing low-frequency components before high-frequency ones during training. The authors provide theoretical foundations for the F-Principle across three distinct training stages: initial, intermediate, and final. Their findings are broadly applicable, covering multilayer networks with diverse activation functions, data population densities, and a wide range of loss functions. This research aims to offer a deeper, quantitative understanding of DNN training dynamics.

Key takeaway

For AI Scientists optimizing deep learning models, understanding the Frequency Principle is crucial. Your training strategies should account for the inherent low-to-high frequency learning bias of DNNs, especially when dealing with complex data or requiring precise high-frequency feature capture. This theoretical foundation suggests that early training phases prioritize broad strokes, while later stages refine details, influencing hyperparameter tuning and early stopping decisions.

Key insights

The paper rigorously establishes the Frequency Principle, explaining how deep neural networks learn low-frequency components before high-frequency ones.

Principles

• DNNs learn low-frequency features first.
• F-Principle applies across training stages.
• Generality holds for various network types.

Method

The paper provides theorems characterizing the F-Principle at initial, intermediate, and final training stages for general DNNs, using proper quantities for each stage.

Topics

• Deep Neural Networks
• Frequency Principle
• Machine Learning Theory
• Training Dynamics
• Activation Functions
• Loss Functions

Best for: Research Scientist, AI Scientist

Related on AIssential

• There Will Be a Scientific Theory of Deep Learning — A scientific theory of deep learning, "learning mechanics," is emerging, focusing on dynamics, aggregate statistics, and quantitative predictions.
• Sequential Fitting: A Different Perspective on the Spectral Bias of Neural Networks — The spectral bias in MLPs stems from "sequential fitting" and "boundary effects," not solely frequency content.
• Frequency-Forcing: From Scaling-as-Time to Soft Frequency Guidance — Explicit coarse-to-fine frequency guidance via a self-sourced, earlier-maturing auxiliary stream improves image generation quality.
• Timing Trick Cuts Energy Used in LLM Training by Up to 14 Percent — Fine-grained, per-kernel DVFS can cut LLM training energy by 14% without speed loss.
• Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential — FMMNNs combine MMNN structure with sine/SinTU activations for superior high-frequency approximation and efficient training.
• Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime — A universal theoretical framework for neural scaling laws and weight spectra in shallow networks is established via sparse estimation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.