Sequential Fitting: A Different Perspective on the Spectral Bias of Neural Networks

2026-06-08 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Research Methodology & Innovation · Depth: Expert, extended

Summary

Conor Rowan and Finn Murphy-Blanchard introduce "sequential fitting" and "boundary effects" as alternative understandings of the spectral bias in neural networks, which describes their tendency to fit low-frequency components before high-frequency ones. They argue that multilayer perceptron (MLP) networks with hyperbolic tangent activations fit target functions by progressing from the domain boundaries inward, building one oscillation at a time. Their analysis in one and two spatial dimensions demonstrates that the target function's behavior near boundaries significantly influences training difficulty, even for functions with nearly identical Fourier spectra. Furthermore, they show that these MLPs iteratively construct smoothed step-like basis functions, a process that contrasts with the oscillatory bases generated by SIREN networks, which eliminate sequential fitting.

Key takeaway

For Machine Learning Engineers working on scientific machine learning applications involving oscillatory or multi-scale functions, you should recognize that neural network regression performance is not solely dictated by frequency content. Your model's ability to fit high-frequency targets can be significantly impacted by the target function's behavior near domain boundaries. Consider employing architectural modifications like SIREN networks or second-order optimization strategies to overcome the "sequential fitting" bias and improve training efficiency for such complex problems.

Key insights

The spectral bias in MLPs stems from "sequential fitting" and "boundary effects," not solely frequency content.

Principles

• Neural networks fit regression targets from low to high frequencies.
• MLP training progresses from domain boundaries inward.
• Target function boundary behavior impacts training difficulty.

Method

The authors demonstrate sequential fitting in 1D and 2D MLP regression using ADAM optimization, analyzing basis function evolution and Fourier spectra.

In practice

• Consider SIREN networks for oscillatory function fitting.
• Use second-order optimizers to mitigate spectral bias.
• Evaluate target function behavior near boundaries.

Topics

• Spectral Bias
• Neural Networks
• Regression Analysis
• Scientific Machine Learning
• Activation Functions
• Basis Functions
• ADAM Optimization

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• JA-SIREN: Deterministic Initialization for Sinusoidal Networks via Spectral Matching — Deterministic initialization via spectral matching for sinusoidal networks significantly enhances INR reproducibility and performance.
• Boundary Variance Inflation Causes Acquisition Bias in Gaussian Processes — Gaussian Process posterior variance inflates at domain boundaries due to kernel truncation, causing acquisition bias independent of objective functions.
• Explaining Data Mixing Scaling Laws — A unified framework explains data mixing scaling laws through capacity competition and noise reduction.
• Statistical Learning Analysis of Physics-Informed Neural Networks — PINNs can be understood as a statistical learning problem minimizing KL divergence with physics as indirect data.
• Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy — LLM gradients exhibit a "spike-tail" anisotropy where dominant linguistic structures suppress long-tail semantic learning.
• A data-driven Fourier-mixture neural-network method for density estimation — A Fourier-trained neural network using Gaussian-Laplace mixtures estimates probability densities from empirical characteristic functions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.