On De-Individuated Neurons: Continuous Symmetries Enable Dynamic Topologies

2023-04-24 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

This paper introduces a novel methodology for dynamic neural networks by utilizing "isotropic activation functions," a class of primitives based on continuous orthogonal symmetries. This approach enables real-time growth (neurogenesis) and shrinkage (neurodegeneration) of network architectures in response to task demands, while maintaining analytical invariance or close approximation of computational function. The core mechanism involves re-expressing typical interconnected layers, such as dense layers and convolutional kernels, through a layer-wise diagonalization procedure. This procedure, facilitated by the basis-independence of isotropic primitives, allows neurons to have one-to-one, ordered connectivity within alternating layers. Inconsequential neurons can be removed, and new inactive "scaffold" neurons added, without significantly degrading network functionality. The method also introduces a tunable "intrinsic length" parameter to ensure analytical invariance during pruning and theoretically demonstrates that isotropic dense networks can asymptotically achieve 50% sparsity while retaining exact functionality. Experimental results on CIFAR10 classification with multilayer perceptrons show smooth transitions between widths with minimal accuracy loss, and suggest that an initial overabundance of neurons followed by pruning can be beneficial, mirroring biological findings.

Key takeaway

For research scientists exploring novel neural network architectures, this work demonstrates a principled approach to dynamic topologies. You should investigate integrating isotropic primitives and layer diagonalization into your models to enable real-time neurogenesis and neurodegeneration. This method offers a path to networks that can adapt their size and connectivity without catastrophic forgetting, potentially leading to more efficient and robust systems, especially when starting with an initially wider network and pruning.

Key insights

Continuous symmetries enable dynamic neural network topologies with real-time growth and pruning while preserving function.

Principles

• Symmetries can prescribe functional forms for neural primitives.
• Basis-independence allows continuous reparameterizations for dynamic topologies.
• Initial overabundance of neurons can lead to better performance.

Method

Diagonalize network layers using singular value decomposition, then prune neurons with singular values below a threshold or add scaffold neurons, adjusting an "intrinsic length" parameter to maintain functional invariance.

In practice

• Implement isotropic activation functions for dynamic architecture adaptation.
• Utilize singular-value based thresholding for neuron pruning.
• Consider starting with larger networks and pruning for improved performance.

Topics

• Isotropic Activation Functions
• Dynamic Neural Architectures
• Network Pruning
• Neurogenesis
• Continuous Symmetries

Best for: Research Scientist, AI Researcher, AI Scientist, Deep Learning Engineer

Related on AIssential

• Hebbian-Oscillatory Co-Learning — HOC-L unifies structural plasticity and phase synchronization in sparse neural networks via synchronization-gated Hebbian learning.
• Preserving Plasticity in Continual Learning via Dynamical Isometry — Dynamical isometry, where layer-wise Jacobian singular values remain near one, is key to preserving plasticity in continual learning.
• Separable neural architectures as a primitive for unified predictive and generative intelligence — Separable Neural Architectures unify diverse models by exploiting factorisable structures for predictive and generative intelligence.
• Robust Parameter and State Estimation in Multiscale Neuronal Systems Using Physics-Informed Neural Networks — PINNs offer robust parameter and state estimation in multiscale neuronal systems, overcoming limitations of traditional methods.
• Neural Networks on Symmetric Spaces of Noncompact Type — A unified point-to-hyperplane distance enables novel neural network architectures on noncompact symmetric spaces.
• Beyond KV-Cache: Test-Time Training & Invariant Latent Topologies for ICL — New methods enable LLMs to learn during inference and generalize across diverse domains by modifying internal weight structures or identifying invariant neural subspaces.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.