Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

2026-05-13 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, medium

Summary

Neural Low-Degree Filtering (Neural LoFi) is a new theoretical framework that models deep neural network learning as an iterative spectral procedure. This approach simplifies gradient-based training dynamics, allowing each layer's learning to decouple and select directions with maximal low-degree correlation to the label. Neural LoFi offers a tractable surrogate mechanism for deep learning, providing a kernel-space interpretation and a framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains concept emergence with given sample complexity, and details how depth constructs new features through low-degree compositionality. Mechanistic experiments on fully connected and convolutional architectures demonstrate that Neural LoFi outperforms lazy random-feature baselines, recovers structured filters, and aligns with early gradient-descent feature discovery on real datasets.

Key takeaway

For research scientists investigating deep learning theory, Neural LoFi offers a mathematically explicit framework to understand hierarchical feature learning beyond the lazy regime. You should consider this spectral theory to predict how representations are formed and how depth contributes to feature construction, potentially guiding the design of more interpretable and efficient deep learning architectures.

Key insights

Neural LoFi models deep learning as an iterative spectral process for hierarchical feature learning.

Principles

• Layer dynamics decouple in the Neural LoFi limit.
• Features are constructed via low-degree compositionality.
• Low-degree correlation guides label selection.

Method

Neural LoFi models deep learning as an iterative spectral procedure where each layer selects directions with maximal accessible low-degree correlation to the label, providing a tractable surrogate mechanism.

In practice

• Analyze representation selection layer by layer.
• Predict concept emergence with sample complexity.
• Recover structured filters in CNNs.

Topics

• Neural Low-Degree Filtering
• Deep Learning Theory
• Hierarchical Feature Learning
• Spectral Theory
• Representation Learning

Best for: Research Scientist, AI Scientist, AI Student

Related on AIssential

• Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning — Neural LoFi explains deep learning's hierarchical feature acquisition via iterative, label-correlated spectral filtering.
• There Will Be a Scientific Theory of Deep Learning — A scientific theory of deep learning, "learning mechanics," is emerging, focusing on dynamics, aggregate statistics, and quantitative predictions.
• From Spatial to Spectral: An Efficient, Frequency-Guided Feature Representation Learner for Small Object Detection — Shifting small object detection from spatial to spectral processing improves efficiency and accuracy by preserving high-frequency features.
• Neural Field Tokenizations with Hierarchy and Spatial Locality Priors — LH-NeF efficiently learns general-purpose neural field representations by integrating locality and hierarchy priors via a single forward pass.
• Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy — LLM gradients exhibit a "spike-tail" anisotropy where dominant linguistic structures suppress long-tail semantic learning.
• DSFNet: Learning Dual-Domain Spectral Operators for Multi-Modality Spatio-Temporal Forecasting in Urban Transportation Systems — DSFNet uses dual-domain spectral filtering to model complex spatio-temporal and cross-modality couplings in urban traffic.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.