Approximation and learning of anisotropic and mixed smooth functions by deep ReLU neural networks

2026-07-27 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Yunfei Yang and Jun Fan's research explores the efficiency of deep ReLU neural networks in approximating and learning smooth functions, specifically anisotropic and mixed smooth Besov spaces. The study establishes an approximation rate of O((WL)-2s) for anisotropic Besov space Bsq,r([0,1]d) with mean smoothness s=(i=1dsi-1)-1, under the embedding condition s>1/q-1/p. For mixed smooth Besov space MBsq,r([0,1]d), an approximation rate of O((WL)-2s) is shown, up to logarithmic factors, for s>1/q-1/p. These findings generalize previous results, demonstrating that sparsity is not always necessary to obtain these rates and providing a precise trade-off between network width (W) and depth (L). The work also derives minimax optimal learning rates for nonparametric regression using these approximation bounds, extending prior analyses to fully-connected networks.

Key takeaway

For AI Scientists and Research Scientists designing deep learning models for high-dimensional data, this work confirms that fully-connected ReLU networks can achieve minimax optimal approximation and learning rates for anisotropic and mixed smooth functions. You should consider the specific smoothness properties of your target functions (e.g., s or s) to precisely tune network width W and depth L. This allows for overcoming the curse of dimensionality in certain contexts, guiding efficient model architecture selection for nonparametric regression tasks.

Key insights

Deep ReLU networks efficiently approximate anisotropic and mixed smooth functions, achieving minimax optimal learning rates.

Principles

• Anisotropic smoothness s dictates approximation rates.
• Mixed smoothness can mitigate the curse of dimensionality.
• Network width and depth trade-off precisely for optimal rates.

Method

Functions are decomposed into piecewise polynomials, then ReLU networks are constructed to approximate these polynomials, handling discontinuities via trifling regions.

In practice

• Apply fully-connected networks for anisotropic Besov functions.
• Use deep composition models for complex function structures.
• Optimize network (W, L) for minimax optimal learning rates.

Topics

• Deep ReLU Networks
• Anisotropic Besov Spaces
• Mixed Smoothness
• Approximation Theory
• Nonparametric Regression
• Curse of Dimensionality

Best for: AI Scientist, Research Scientist

Related on AIssential

• Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networks — ReLU neural networks can approximate complex decision regions in o-minimal structures, yielding quantitative approximation and statistical learning rates.
• Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks — A variational formulation for shallow neural networks allows global optimization via a linear system, bypassing iterative training.
• Layer-wise Geometric Approximation Rates for Deep Networks — Deep network depth can quantitatively refine function approximation layer-wise, enabling adaptive, nested architectures.
• On the Principles of Deep Feedforward ReLU Networks — The paper reveals the "black box" of deep feedforward ReLU networks by explaining back-propagation solutions through the concept of "paths."
• Deep regression learning from dependent observations with minimum error entropy principle — MEE-based deep neural networks achieve minimax optimal rates for nonparametric regression with dependent data.
• New Complexity-Theoretic Frontiers of Tractability for Neural Network Training — New algorithms expand polynomial-time tractability for optimal training of specific linear and ReLU neural network architectures.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.