Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

2026-06-16 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This work introduces novel activation functions enabling arbitrary-accuracy Sobolev approximation by fixed-size neural networks, addressing a gap in controlling weak derivatives. The Elementary Universal Activation Function ("EUAF") allows approximation of W^2,inf((a,b)^d) functions in the W^1,inf-norm with a fixed-size network of width 4^d(5d^2+8d+3) and depth 2d+5. To extend this to higher-order Sobolev spaces, the authors present the Differentiable Universal Activation Function ("DUAF") family, including the C^inf "DUAF"_inf. These achieve W^s-1,inf-norm approximation for W^s,inf((a,b)^d) functions with fixed width N_s,d=O(4^d(s+d)C(s+d-1,d)) and depth L_s,d=O(s+d). Sigmoidal variants, "DUAF"_n, also provide fixed-size W^s-1,inf approximation for 1<=s<=n, with width S_n N_s,d and depth (3n^2+5)L_s,d. For structured Sobolev targets, "DUAF"_inf networks achieve reduced complexity, with width O(dQ s^2) and depth O(s).

Key takeaway

For research scientists designing neural networks for scientific computing or PDE-based models, this work fundamentally changes the approach to Sobolev approximation. You can now achieve arbitrary accuracy in Sobolev norms with fixed-size networks by employing specialized activation functions like "EUAF" or "DUAF"_inf. This eliminates the need for ever-growing network sizes to improve derivative control. Consider implementing these activations, especially for structured problems where network width scales linearly with channels and dimension.

Key insights

Fixed-size neural networks can achieve arbitrary-accuracy Sobolev approximation using specially designed activation functions.

Principles

• Sobolev norms are crucial for PDE-based models, controlling function values and derivatives.
• Super-expressive activations enable fixed-size networks for Sobolev approximation.
• Activation function smoothness can be explicitly prescribed for higher-order Sobolev approximation.

Method

The method involves local Taylor approximation, encoding coefficients via staircase and point-fitting subnets, and gluing local approximants using smooth, fixed-size gates.

In practice

• Use "EUAF" for W^1,inf approximation of W^2,inf functions.
• Employ "DUAF"_inf for C^inf Sobolev approximation.
• Leverage structured Sobolev targets for reduced network width.

Topics

• Sobolev Approximation
• Fixed-Size Neural Networks
• Universal Activation Functions
• DUAF
• PDE Solutions
• Network Expressivity

Best for: AI Scientist, Research Scientist

Related on AIssential

• Certified and accurate computation of function space norms of deep neural networks — Certified function space norms for neural networks are achieved via interval arithmetic and adaptive quadrature.
• Weighted universal approximation of differentiable maps on infinite-dimensional manifolds — The universal approximation theorem for FNNs is generalized to differentiable maps, including derivative approximation, via a weighted Nachbin theorem.
• Continuous-time reinforcement learning: ellipticity enables model-free value function approximation — Ellipticity in continuous-time diffusions enables stable, model-free RL with function approximation, akin to supervised learning.
• Learning functional components of PDEs from data using neural networks — Neural networks can approximate unknown functional components within PDEs by embedding and training them with data.
• Towards Unified Native Spaces in Kernel Methods — A new unified kernel class integrates diverse positive definite kernels, enabling characterization of associated Sobolev spaces.
• The 60-Year Hunt for AI's Most Important Function — Activation functions are critical for neural network expressiveness, evolving from simple thresholds to complex gated, smooth functions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.