Fourier fractal dimension to predict the generalization of deep neural networks

2026-06-06 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

A novel generalization measure for deep neural networks is proposed, based on the Fourier fractal dimension of the network's weight variations. This approach analyzes the characteristic function of Lévy-driven stochastic differential equations in the frequency domain to extract a metric capturing the geometric complexity of the learning process. The authors also introduce a customized Fourier-based optimizer designed to actively regularize this fractal dimension during training. Empirical evaluations on CIFAR-10, SVHN, and MNIST datasets demonstrate a strong correlation between the proposed Fourier generalization measure and the actual generalization gap. The method achieves leading Kendall rank correlation coefficients, surpassing existing norm-based, margin-based, and PAC-Bayesian measures.

Key takeaway

For Machine Learning Engineers optimizing deep neural networks, you should consider integrating Fourier fractal dimension analysis. This approach offers a robust, hold-out-free method to predict generalization performance, potentially streamlining model selection and hyperparameter tuning. Implementing the proposed Fourier-based optimizer could also actively regularize network complexity, leading to more stable and generalizable models. You can achieve leading correlation with actual generalization gaps.

Key insights

Fourier fractal dimension of network weight variations predicts deep neural network generalization without hold-out data.

Principles

• SGD dynamics induce complex, scale-invariant trajectories.
• Frequency-domain fractal analysis predicts model generalizability.
• Regularizing fractal dimension improves optimization stability.

Method

Proposes a Fourier generalization measure by analyzing characteristic functions of Lévy-driven SDEs in the frequency domain, then introduces a Fourier-based optimizer for active regularization.

In practice

• Apply Fourier fractal dimension for generalization prediction.
• Implement Fourier-based optimizers for regularization.
• Evaluate against CIFAR-10, SVHN, MNIST datasets.

Topics

• Deep Neural Networks
• Generalization Prediction
• Fourier Fractal Dimension
• Stochastic Gradient Descent
• Optimization Algorithms
• Lévy Processes

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Generalization at the Edge of Stability — Generalization at the "edge of stability" is linked to the "sharpness dimension" of chaotic optimization attractors.
• ConvNeXt-FD: A Fractal-Based Deep Model for Robust Biomedical Image Segmentation — ConvNeXt-FD enhances biomedical image segmentation by combining ConvNeXt with fractal-inspired boundary regularization for superior shape fidelity.
• Pointwise Generalization in Deep Neural Networks — Pointwise Riemannian Dimension explains deep network generalization by characterizing feature representation complexity.
• A data-driven Fourier-mixture neural-network method for density estimation — A Fourier-trained neural network using Gaussian-Laplace mixtures estimates probability densities from empirical characteristic functions.
• Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime — The paper proposes an equivalent Wishart Ansatz and kernel renormalization to predict Bayesian DNN generalization in the proportional limit.
• Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime — A universal theoretical framework for neural scaling laws and weight spectra in shallow networks is established via sparse estimation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.