Expressivity of Shallow Neural Networks Over Finite Fields

2026-07-21 · Source: cs.NE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

The paper "Expressivity of Shallow Neural Networks Over Finite Fields" investigates the expressivity of shallow polynomial neural networks (PNNs) with monomial activation functions over finite fields. It defines a "neuromanifold" as the image of the network weights mapping into polynomial rings and quantifies expressivity by its cardinality, deriving natural lower and upper bounds. The work highlights critical differences in neuromanifolds between characteristic zero and finite-characteristic fields, showing, for instance, that for architecture δ=(2,2,2) with r=2, the neuromanifold over finite fields has approximately half the cardinality of the ambient space, unlike its Zariski open complex counterpart. This research connects to counting rational points over finite fields, a problem linked to Weil conjectures, and is motivated by practical advantages of weight quantization.

Key takeaway

For AI Scientists and Research Scientists developing quantized neural networks, this work reveals that the choice of finite field characteristic profoundly impacts network expressivity. You should account for these algebraic differences, especially when designing shallow polynomial neural networks, as their neuromanifold cardinality can be significantly constrained compared to complex-valued counterparts. This understanding is crucial for optimizing energy consumption and storage in practical deployments.

Key insights

Finite field characteristics critically alter shallow polynomial neural network expressivity and neuromanifold cardinality compared to complex fields.

Principles

• Expressivity is quantifiable by neuromanifold cardinality over finite fields.
• Field characteristic significantly impacts neuromanifold structure and size.
• Weight quantization benefits from understanding finite field expressivity.

Method

The method involves defining neuromanifolds as images of parameter maps, quantifying expressivity via cardinality, and deriving bounds by counting rational points over finite fields, also using projective space analysis.

In practice

• Consider finite field characteristics when designing quantized neural networks.
• Use combinatorial counting to assess PNN expressivity for specific architectures.

Topics

• Polynomial Neural Networks
• Finite Fields
• Network Expressivity
• Neuromanifolds
• Weight Quantization
• Algebraic Geometry

Best for: AI Scientist, Research Scientist

Related on AIssential

• An Algebraic View of the Expressivity of Recurrent Language Models — Arithmetic semantics fundamentally determines recurrent neural network expressivity and recognized formal languages.
• A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks — Functional-space mean-field theory extends neural network analysis to three-layer models, proving convergence and feature learning.
• Expressivity of congruence-based architectures for DNNs on positive-definite matrices — Semi-orthogonality in congruence layers limits DNN expressivity for SPD matrix classification due to spectral diversity loss.
• Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation — Neural network complexity is governed by algorithmic complexity, not just regularity, viewing NNs as computational models.
• Spectral Higher-Order Neural Networks Have Sharp Expressivity Bounds — Spectral Higher-Order Neural Networks (SHONNs) use spectral attributes and weight sharing to overcome neural hypergraph parameter explosion.
• Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations — Constraining neural network expressibility to an algebraic variety eliminates grokking, leading to binary representability outcomes.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.NE updates on arXiv.org.