Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

2026-07-08 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This paper introduces a unified function space theory for deep fully connected neural networks, defining functions recursively as L1-bounded linear combinations of activated functions. Unlike prior work focused on homogeneous activations like ReLU, this framework accommodates a broad range of homogeneous and non-homogeneous activation functions. It unifies existing norm-based complexity bounds and variational depth characterizations, enabling novel analyses of representable functions. A key finding is a "depth saturation" result for univariate ReLU networks, demonstrating that increased depth provides only a small constant rescaling of the function class, without adding functional diversity. Consequently, deep norm-controlled ReLU functions in any dimension cannot exhibit high frequencies along any direction. This suggests that commonly cited expressivity benefits of depth diminish when network complexity is controlled by an appropriate function space norm, rather than parameter count.

Key takeaway

For research scientists evaluating deep network architectures or designing regularization strategies, you should critically re-evaluate how network complexity is measured. This work suggests that depth's expressivity benefits, especially for high-frequency functions, are often an artifact of compounded layerwise rescaling rather than inherent nonlinearity. Prioritize regularization that controls function space norms over simple parameter counts to gain a more accurate understanding of a deep network's true functional capacity.

Key insights

A unified function space theory reveals depth's limited functional expressivity when complexity is controlled by true function norms, not parameter counts.

Principles

• Network complexity should be controlled by function space norms, not parameter counts.
• Depth's expressivity for high frequencies often stems from compounded layerwise rescaling.
• Deep neural variation spaces unify complexity notions for diverse activation functions.

Method

Deep neural variation spaces (mathcal{V}_L) are recursively constructed using L1-bounded linear combinations of normalized activations (sigma_s(t) := sigma(st)/s) from preceding layers.

In practice

• Complexity bounds for mathcal{B}_L extend to classes from SOSW and R_Ba representation costs.
• Norm-penalized data fitting problems yield width-bounded neural network solutions.

Topics

• Deep Neural Networks
• Function Space Theory
• Network Complexity
• Activation Functions
• Representer Theorem
• Depth Saturation

Best for: AI Scientist, Research Scientist

Related on AIssential

• Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks — A new framework unifies representation costs and native function spaces for parametric data-fitting methods, revealing deep neural network inductive biases.
• A Function-Centric Perspective on Flat and Sharp Minima — Sharpness in neural network minima is function-dependent, not solely indicative of poor generalization.
• Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation — Neural network complexity is governed by algorithmic complexity, not just regularity, viewing NNs as computational models.
• Certified and accurate computation of function space norms of deep neural networks — Certified function space norms for neural networks are achieved via interval arithmetic and adaptive quadrature.
• Pointwise Generalization in Deep Neural Networks — Pointwise Riemannian Dimension explains deep network generalization by characterizing feature representation complexity.
• Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds — Compositionality in neural networks arises only within a precise, narrow regime of network depth and connectivity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.