Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

2026-06-12 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

The Fourier Multi-Component and Multi-Layer Neural Network (FMMNN) is introduced as an effective architecture for capturing high-frequency components in function approximation. This network integrates the Multi-Component and Multi-Layer Neural Network (MMNN) structure with sine or Sine Truncated Unit (SinTU) activation functions. FMMNNs theoretically exhibit exponential expressive power, achieving $O(N^{-L})$ approximation errors for 1-Lipschitz functions. This notably improves upon $O(2^{-\sqrt{L}})$ for FCNNs. Their optimization landscape is also significantly more favorable than FCNNs. Numerical experiments confirm FMMNNs consistently provide superior accuracy and efficiency across diverse tasks. They particularly excel for high-frequency or non-smooth functions, demonstrating enhanced stability to training hyperparameters.

Key takeaway

For machine learning engineers and AI scientists working on complex function approximation, you should evaluate Fourier Multi-Component and Multi-Layer Neural Networks (FMMNNs). This architecture, especially with sine or SinTU activations, offers superior accuracy and training stability for high-frequency or non-smooth data. It outperforms traditional FCNNs, even with challenging hyperparameters. Integrating FMMNNs could significantly improve your model performance and convergence for these difficult problems.

Key insights

FMMNNs combine MMNN structure with sine/SinTU activations for superior high-frequency approximation and efficient training.

Principles

• MMNNs decompose complex functions and compose layers for efficient training.
• Sine and SinTU activations significantly enhance high-frequency approximation.
• Fixing "W" and "b" parameters in MMNN layers simplifies optimization.

Method

MMNNs structure layers as linear combinations of randomized hidden neurons. Only "A" and "c" parameters are updated, while "W" and "b" are fixed post-initialization.

In practice

• Employ FMMNNs with sine or SinTU for tasks involving high-frequency components.
• Utilize SinTU activations for approximating non-smooth functions with singularities.

Topics

• Fourier Neural Networks
• MMNN Architecture
• Sine Activation
• SinTU
• Function Approximation
• High-Frequency Learning
• Optimization Landscape

Code references

• ShijunZhangMath/FMMNN

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Fourier Neural Operators with rank-1 lattice points and hyperbolic cross — Replacing FNO's tensor product grids with rank-1 lattices and hyperbolic cross improves efficiency and accuracy using fewer resources.
• Multi-Rate Mixture of Experts for Accelerating Liquid Neural Network Training — Multi-Rate Mixture-of-Experts enhances Liquid Neural Networks for complex, multi-scale time-series modeling.
• Frequency-Enhanced Dual-Subspace Networks for Few-Shot Fine-Grained Image Classification — FEDSNet improves few-shot fine-grained image classification by integrating frequency domain features to enhance structural stability and reduce overfitting.
• The 60-Year Hunt for AI's Most Important Function — Activation functions are critical for neural network expressiveness, evolving from simple thresholds to complex gated, smooth functions.
• PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training — Polynomial preconditioning reshapes weight singular-value spectra to improve LLM training stability and efficiency without inference overhead.
• Ilya’s List, Part 17: Relational Recurrent Neural Networks: What If an RNN Had More Than One… — Relational Recurrent Neural Networks enhance memory by using multiple interacting slots instead of a single, undifferentiated state vector.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.