Statistics of correlations in nonlinear recurrent neural networks

2026-04-23 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This research introduces a path-integral framework to precisely calculate correlation statistics in nonlinear recurrent neural networks (RNNs), including systematic 1/N corrections for large networks. The method generalizes prior work on linear networks by incorporating diverse nonlinear activation functions as interaction terms within the path integral, which resolves the instability issues of linear theory and ensures a strictly positive participation dimension. The approach simplifies the network's stochastic dynamics into a few collective variables, enabling efficient computation of correlation functions. Explicit results are presented for power-law and a new class of Padé approximant activation functions, demonstrating scaling behavior controlled by network coupling and excellent agreement with numerical simulations, even for networks with a few hundred neurons.

Key takeaway

For AI scientists and research scientists modeling complex neural systems, this work provides a robust analytical framework for understanding nonlinear RNN dynamics. You should consider adopting path-integral methods to derive exact correlation statistics, especially when dealing with nonlinear activation functions and finite-size effects. This approach offers a more stable and comprehensive understanding of network behavior, including the critical participation dimension, compared to traditional linear models.

Key insights

A path-integral framework accurately computes correlation statistics in nonlinear RNNs, resolving linear model instabilities.

Principles

• Nonlinear activation functions stabilize network dynamics.
• Collective variables capture full correlation statistics.
• 1/N corrections are crucial for participation dimension.

Method

The method uses a path-integral representation of stochastic dynamics, reducing the description to collective variables, and applies a large N expansion with saddle point approximation to derive correlation functions and participation dimension.

In practice

• Use Padé approximants for tractable nonlinear activation functions.
• Simulate networks for 10-20 synaptic time constants for convergence.
• Consider 1/N corrections for accurate cross-correlation analysis.

Topics

• Recurrent Neural Networks
• Correlation Statistics
• Path-Integral Methods
• Nonlinear Activation Functions
• Participation Dimension

Best for: AI Scientist, Research Scientist

Related on AIssential

• Recurrent Neural Networks (RNNs) - Explained — RNNs utilize a hidden state as a compressed memory, enabling sequential data processing by looping output back as input.
• Characterizing the Discrete Geometry of ReLU Networks — The connectivity graph of ReLU network linear regions has a bounded average degree and diameter, independent of network size.
• Theory of learning of high-dimensional controlled non-linear dynamical systems (I): models and methods — Dynamical Mean Field Theory (DMFT) can exactly solve coupled inference and training dynamics in high-dimensional Neural ODEs.
• A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention — A unified theory determines critical inverse-temperature scaling for self-attention based on a gap-counting function.
• Interpreting FCDNNs via RG on Exponential Family — The training of FCDNNs is equivalent to Renormalization Group calculations, explaining their feature extraction and performance.
• Chaotic Oscillator Networks for Classification Tasks — Machine learning can approximate coupling terms in chaotic oscillator networks for scalable, efficient classification and pattern recognition.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.