RepNet: Tackling spectral bias in deep neural networks via parameter reparameterization

2026-06-15 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

RepNet is a reparameterized deep neural network (DNN) model developed to mitigate spectral bias, a common limitation in DNNs when capturing oscillatory and multiscale behaviors in scientific computing. This study identifies that the initial slope scale and the distribution of partition points are crucial for resolving rapid oscillations, particularly in shallow ReLU networks. RepNet addresses this by reparameterizing the weights and biases within the first hidden layer of ReLU and tanh networks. This mechanism allows for effective control over the initial slope scale and provides an appropriate distribution of initial partition points, enabling adaptive frequency scaling during training. Quantitative estimates for output and slope magnitudes are also derived to guide initialization. Numerical experiments, including multiscale one- and four-dimensional function approximation, forward and inverse PDE problems with physics-informed neural networks (PINNs), and operator learning, demonstrate RepNet's improved accuracy in capturing highly oscillatory features with only slightly additional computational cost compared to vanilla DNNs.

Key takeaway

For Machine Learning Engineers developing DNNs for scientific computing tasks involving highly oscillatory or multiscale data, RepNet offers a robust solution to overcome spectral bias. You should consider integrating RepNet's reparameterization approach into your ReLU and tanh networks to achieve significantly improved accuracy in capturing complex features. This method provides adaptive frequency scaling during training with only slightly additional computational cost, making it a practical enhancement for challenging problems like PDE solving or operator learning.

Key insights

RepNet tackles DNN spectral bias in high-frequency data by reparameterizing first-layer weights for adaptive frequency scaling.

Principles

• Spectral bias limits DNNs in high-frequency tasks.
• Initial slope scale and partition points are key for oscillations.
• Adaptive frequency scaling improves DNN performance.

Method

RepNet reparameterizes first hidden layer weights and biases in ReLU/tanh networks. This controls initial slope scale and partition points, enabling adaptive frequency scaling during training, guided by quantitative estimates.

In practice

• Improve multiscale function approximation.
• Enhance PINN accuracy for PDE problems.
• Boost operator learning for oscillatory features.

Topics

• RepNet
• Spectral Bias
• Deep Neural Networks
• Parameter Reparameterization
• Multiscale Problems
• Physics-Informed Neural Networks
• Operator Learning

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• The Reparameterization Trick in 60 Seconds — The Reparameterization Trick enables gradient-based learning through stochastic sampling by isolating randomness.
• Sequential Fitting: A Different Perspective on the Spectral Bias of Neural Networks — The spectral bias in MLPs stems from "sequential fitting" and "boundary effects," not solely frequency content.
• Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics — Gradient descent in ReLU networks can be modeled as collective dynamics in training-set space, mediated by layer-wise Gram operators.
• Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training — Forward-Forward learning's real-data scaling and memory efficiency are significantly overstated by synthetic benchmarks, revealing limitations compared to backpropagation.
• Hard-Won Lessons from Training a Very Deep GAN — Wasserstein loss and careful regularization are crucial for stable, deep GAN training.
• Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy — LLM gradients exhibit a "spike-tail" anisotropy where dominant linguistic structures suppress long-tail semantic learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.