Axiomatizing Neural Networks via Pursuit of Subspaces

2026-05-21 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, long

Summary

The Pursuit of Subspaces (PoS) hypothesis introduces an axiomatic framework to geometrically explain deep neural network behavior, addressing their "black box" nature. This framework, developed by Mehmet Yamaç and colleagues from Tampere University, Qatar University, and Radboud University, formulates neural network operations through a set of geometric postulates. PoS introduces four geometric axioms that describe how deep networks learn compact data representations, offering mathematically grounded explanations for generalization, hallucination control, and stability. It generalizes Sparse Representation into differential geometry, extending single-layer continuous piecewise-linear models to hierarchical curved-space representations. The framework interprets nonlinear activations (like ReLU) as angular selectors, residual connections as normal-space component annihilation, and attention mechanisms as collaborative residual-removal procedures. Experimental validation includes zero-shot anomaly detection, the PoS Former architecture, and image restoration.

Key takeaway

For AI Scientists and Research Scientists focused on demystifying neural network behavior and designing explainable architectures, the Pursuit of Subspaces (PoS) framework offers a principled geometric foundation. You should consider PoS's axiomatic approach to understand how mechanisms like ReLU, residual connections, and attention enforce compact representations. This perspective can guide the development of novel, inherently explainable deep learning models and improve generalization and stability.

Key insights

The PoS hypothesis provides a geometric, axiomatic framework for neural networks, unifying representation, computation, and generalization through subspace pursuit.

Principles

• Neural networks learn compact data representations via geometric axioms.
• Orthogonality and disentanglement emerge as necessary conditions for stable projections.
• Selective suppression of residual directions enforces compact representations.

Method

The PoS framework models representations as unions of low-dimensional smooth submanifolds with structured projection operators, inducing transversal decomposition of tangent spaces.

In practice

• PoS explains ReLU as angular selectors.
• Residual connections refine representations.
• Attention aligns representations via shared geometry.

Topics

• Neural Network Axiomatization
• Pursuit of Subspaces
• Geometric Deep Learning
• Sparse Representation Theory
• Manifold Hypothesis
• Explainable AI

Best for: AI Scientist, Research Scientist

Related on AIssential

• Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks — A new framework unifies representation costs and native function spaces for parametric data-fitting methods, revealing deep neural network inductive biases.
• Asymptotic Signal Subspace Recovery in Softmax Attention Models — Softmax attention mechanisms can rigorously extract hidden signals from high-dimensional noise through gradient-based learning dynamics.
• Generalization in Nonlinear Least Squares via Learned Feature Geometry — New generalization bounds for nonlinear least-squares models depend on learned feature geometry and data-dependent effective dimension, not just parameter count.
• The New Geometry of Intelligence #ai — Neural networks under capacity pressure form a "crystalline latent structure" using "tight frames" for efficient feature representation.
• Sensor-Conditioned Representation Learning via Scene-Relevant Observation Quotients — Sensor-conditioned representations require evaluation beyond predictive utility, focusing on preserving sensing-justified scene distinctions in their latent geometry.
• Neural-Guided Domain Restriction to Accelerate Pseudospectra Computation for Structured Non-normal Banded Matrices — A neural network predicts sensitive regions to accelerate pseudospectra computation for non-normal matrices.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.