Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks

2026-06-16 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Greg Ongie and Rahul Parhi introduce a general framework for analyzing representation costs in parametric data-fitting methods, focusing on their parameter-space regularizers. This framework defines representation costs for arbitrary parametric models and reveals their induced native function spaces, unifying existing function-space perspectives. The authors prove that representer theorems hold within this abstract setting and rigorously connect parametric methods to equivalent nonparametric descriptions under sufficient overparameterization. Classical techniques like kernel methods, wavelets, and shallow neural networks, with their respective native spaces (reproducing kernel Hilbert spaces, Besov spaces, variation spaces), emerge as specific instances. A significant new finding is that for depth-L feedforward ReLU networks, their induced native spaces are p-normable quasi-Banach spaces where p = 2/L, indicating that deep neural network inductive bias for depths L > 2 cannot be captured by standard norms.

Key takeaway

For AI scientists developing or analyzing deep learning models, understanding the mathematical foundations of representation costs is crucial. This research reveals that for depth-L feedforward ReLU networks, their inductive bias is characterized by p-normable quasi-Banach spaces with p = 2/L, implying standard norms cannot capture this bias for L > 2. You should consider these native function space properties when designing regularization strategies or interpreting model generalization, especially for deeper architectures.

Key insights

A new framework unifies representation costs and native function spaces for parametric data-fitting methods, revealing deep neural network inductive biases.

Principles

• Parametric methods induce native function spaces.
• Representer theorems apply in abstract settings.
• Deep NN inductive bias is p-normable quasi-Banach.

Method

The framework analyzes representation costs via parameter-space regularizers, defining costs for arbitrary parametric models and revealing their induced native function spaces, unifying existing views.

In practice

• Analyze inductive bias of deep ReLU networks.
• Understand function spaces of parametric models.
• Connect kernel methods to abstract frameworks.

Topics

• Representation Costs
• Parametric Models
• Deep Neural Networks
• Quasi-Banach Spaces
• Functional Analysis
• Inductive Bias

Best for: Research Scientist, AI Scientist

Related on AIssential

• Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective — Neural network generalization is geometrically characterized by the Representation Gap, scaling with intrinsic data dimension and model equivariance.
• The Mathematical Foundations of Intelligence [Professor Yi Ma] — Intelligence can be formalized through parsimony and self-consistency, enabling principled AI architecture design.
• Certified and accurate computation of function space norms of deep neural networks — Certified function space norms for neural networks are achieved via interval arithmetic and adaptive quadrature.
• Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics — Gradient descent in ReLU networks can be modeled as collective dynamics in training-set space, mediated by layer-wise Gram operators.
• Neural Field Tokenizations with Hierarchy and Spatial Locality Priors — LH-NeF efficiently learns general-purpose neural field representations by integrating locality and hierarchy priors via a single forward pass.
• Axiomatizing Neural Networks via Pursuit of Subspaces — The PoS hypothesis provides a geometric, axiomatic framework for neural networks, unifying representation, computation, and generalization through subspace pursuit.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.