Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations

2026-07-15 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

This study investigates grokking in neural networks by examining a limiting case where the architecture's expressible function class collapses to a finite-dimensional algebraic variety. Researchers used two-layer networks with a holomorphic monomial activation sigma(z)=z^k, trained on modular arithmetic tasks encoded via roots of unity. They found that the network output is confined to a (k+1)-dimensional subspace of characters of (Z_p)^2, representing an O(k/p^2) slice of the full function space. A task is representable if its discrete Fourier support lies on the diagonal u+v = k (mod p), or m+n=k for linear-phase targets. Non-representable targets cannot be fit, even on the training set, resulting in a positive lower bound on training loss, independent of width. Across 585 runs, this algebraic prediction matched observed outcomes with 99.8% accuracy, showing instant success or outright failure, with no memorization or grokking. This binary behavior represents the extreme end of the capacity-grokking relationship.

Key takeaway

For AI scientists designing or debugging neural networks, understanding the algebraic limits of your model's expressibility is crucial. This research demonstrates that severe architectural constraints can eliminate grokking, forcing a binary outcome of instant success or outright failure. If your network struggles with a task, consider whether its expressible function class can even represent the target, rather than solely focusing on training dynamics or capacity. This perspective shifts debugging from "why isn't it grokking?" to "can it represent this at all?".

Key insights

Constraining neural network expressibility to an algebraic variety eliminates grokking, leading to binary representability outcomes.

Principles

• Algebraic confinement can prevent memorization and generalization.
• Grokking behavior is a function of network capacity and expressibility.
• Representability can be algebraically characterized for specific network types.

Method

The study used two-layer networks with sigma(z)=z^k holomorphic monomial activation, trained on modular tasks encoded via roots of unity, to analyze algebraic representability.

In practice

• Evaluate task representability against network's algebraic constraints.
• Consider activation functions that limit function class expressibility.

Topics

• Grokking
• Neural Network Theory
• Modular Arithmetic
• Algebraic Representability
• Holomorphic Activations
• Model Capacity

Best for: Research Scientist, AI Scientist

Related on AIssential

• Expressivity of Shallow Neural Networks Over Finite Fields — Finite field characteristics critically alter shallow polynomial neural network expressivity and neuromanifold cardinality compared to complex fields.
• At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics — Network embedding compression significantly lags generalization, overstating effective rank at grokking transition.
• Structural Preservation and the Logical Expressiveness of Graph Neural Networks — GNN classifier expressiveness is characterized by graded modal logic fragments based on structural preservation properties.
• Constraining the outputs of ReLU neural networks — Algebraic varieties and polynomial constraints reveal fundamental structural limitations and expressive properties of ReLU networks.
• The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior — Grokking in arithmetic transformers is primarily a decoder readout bottleneck, not an encoder learning deficiency.
• Why Do Neural Networks Actually Work? Four Questions Nobody Has Fully Answered — Understanding neural networks requires addressing four distinct, partially answered questions rather than a single unified explanation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.