On Explicit Super-Expressive Approximation for Neural Networks

2026-07-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This work introduces a novel fixed-architecture neural network approximation framework that quantitatively characterizes parameter magnitude relative to approximation error, addressing a gap in prior super-expressive approximation research. Utilizing the Chinese Remainder Theorem (CRT) as a constructive encoding mechanism, the authors demonstrate explicit parameter-error trade-offs. For Lipschitz continuous functions on [0,1]^D, they construct a network of width max{D,4} and depth 5 with parameter magnitude log_2P=O(ε^(-2D)log(1/ε)). For Hölder-smooth functions in C^(r,γ)_A([0,1]^D), a network of width max{2D, D+5N+1} and depth r+9 achieves log_2P=O(ε^(-2D/(r+γ))log(1/ε)), showing that higher smoothness reduces the complexity exponent. This approach offers a dual perspective to traditional approximation, where network size typically grows exponentially with decreasing error.

Key takeaway

For AI Scientists exploring neural network approximation theory, this research offers a critical quantitative understanding of fixed-architecture super-expressiveness. Your decision on network design can now consider the explicit trade-off between approximation error and parameter magnitude, especially when targeting Lipschitz or Hölder-smooth functions. This work suggests that for specific function classes, fixing network size is theoretically viable, shifting the complexity to parameter encoding rather than architectural scaling.

Key insights

Fixed-architecture neural networks can achieve super-expressive approximation with explicit parameter-error trade-offs using the Chinese Remainder Theorem.

Principles

• Super-expressive approximation can fix network architecture by transferring approximation burden to parameter magnitude.
• Higher function smoothness (Hölder-smoothness) can mitigate the growth of required parameter magnitude.
• The Chinese Remainder Theorem enables explicit encoding of quantized function values for fixed-architecture networks.

Method

Quantize function values and assign unique coprime integers to domain grids. The Chinese Remainder Theorem then explicitly links grids to quantized values, enabling reconstruction of the target function or a gridwise polynomial surrogate using elementary activations.

Topics

• Neural Network Approximation
• Fixed-Architecture Networks
• Chinese Remainder Theorem
• Super-Expressive Activations
• Lipschitz Functions
• Hölder-Smooth Functions
• Parameter Magnitude Bounds

Best for: Research Scientist, AI Scientist

Related on AIssential

• On Explicit Super-Expressive Approximation for Neural Networks — The Chinese Remainder Theorem enables explicit parameter bounds for super-expressive fixed-architecture neural networks.
• Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation — Neural network complexity is governed by algorithmic complexity, not just regularity, viewing NNs as computational models.
• Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective — Transformers can overcome the curse of dimensionality, demonstrating strong expressive capabilities for function approximation.
• Compositional Approximation Can Strictly Outperform Superpositional Approximation — Compositional approximation can strictly outperform superpositional methods for certain function classes, achieving arbitrarily larger approximation rate gaps.
• Expressivity of Shallow Neural Networks Over Finite Fields — Finite field characteristics critically alter shallow polynomial neural network expressivity and neuromanifold cardinality compared to complex fields.
• A "Lay" Introduction to "On the Complexity of Neural Computation in Superposition" — Neural networks achieve representational efficiency through superposition, with specific computational complexity bounds.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.