Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks

2026-06-12 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new general framework analyzes representation costs for parametric data-fitting methods by examining their parameter-space regularizers. This approach defines representation costs for arbitrary parametric models and reveals their induced native function spaces, unifying existing function-space views. The framework proves representer theorems for parametric methods on these native spaces and rigorously connects parametric and nonparametric descriptions under sufficient overparameterization. It encompasses classical methods like kernel methods (reproducing kernel Hilbert spaces) and wavelets (Besov spaces). Crucially, for depth-$L$ feedforward ReLU networks, the framework shows their induced native spaces are $p$-normable quasi-Banach spaces with $p = 2/L$, indicating that deep neural network inductive bias cannot be captured by norms for depths $L > 2$.

Key takeaway

For AI scientists researching deep learning theory, this framework offers a rigorous foundation for understanding inductive biases. You should consider how network depth $L$ directly impacts the native function space, particularly for ReLU networks where $p=2/L$ implies that standard norms cannot capture the inductive bias for $L > 2$. This insight is critical when designing architectures or interpreting model generalization capabilities.

Key insights

A framework defines representation costs and native function spaces for parametric models, revealing deep neural network inductive biases.

Principles

• Representation costs define native function spaces.
• Overparameterization links parametric and nonparametric methods.
• Deep network inductive bias is not always norm-capturable.

Method

The framework analyzes parametric data-fitting methods via parameter-space regularizers to define representation costs and identify induced native function spaces, unifying existing views.

Topics

• Representation Costs
• Parametric Models
• Deep Neural Networks
• Quasi-Banach Spaces
• Inductive Bias
• Functional Analysis

Best for: Research Scientist, AI Scientist

Related on AIssential

• Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks — A new framework unifies representation costs and native function spaces for parametric data-fitting methods, revealing deep neural network inductive biases.
• Convolutional Neural Networks (CNNs) - Explained — CNNs use local kernels and pooling to preserve spatial relationships and extract hierarchical features from image data.
• Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective — Neural network generalization is geometrically characterized by the Representation Gap, scaling with intrinsic data dimension and model equivariance.
• How does feature learning reshape the function space? — Feature learning in two-layer neural networks deforms function space via a data-adaptive, target-dependent kernel.
• Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions — Transformers struggle with sensitive functions because their parameter space geometry makes these functions nearly impossible to find via random initialization.
• The Mathematical Foundations of Intelligence [Professor Yi Ma] — Intelligence can be formalized through parsimony and self-consistency, enabling principled AI architecture design.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.