Recurrent neural networks approximate continuous functions

2026-06-18 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

A new study demonstrates that recurrent neural networks (RNNs) can approximate any continuous function on [-1,1] to arbitrary precision using a single, fixed network. Unlike classical approximation theorems requiring a new network for improved accuracy, this research proves that a ReLU RNN with fixed weights and hidden dimension can achieve this by simply running for a longer duration. The underlying mechanism involves a novel intermediate model, the Turing machine with neural units (TMNU), which combines algorithmic flexibility with the rigidity needed for RNN simulation. This approach yields convergence rates reflecting polynomial approximation rates. The authors also establish minimax lower bounds, confirming that runtime is an essential resource in this fixed-network approximation paradigm, not merely a theoretical artifact.

Key takeaway

For research scientists exploring efficient neural network architectures, this finding suggests a paradigm shift: you might not need to retrain or redesign an RNN for higher accuracy. Instead, consider designing fixed-weight ReLU RNNs that achieve improved approximation by simply extending their runtime. This approach could simplify model deployment and adaptation, particularly in systems requiring dynamic precision adjustments without architectural changes. Your focus could shift from network complexity to optimizing runtime for desired accuracy levels.

Key insights

A single, fixed ReLU RNN can approximate any continuous function by running longer, enabled by a Turing machine with neural units.

Principles

• Fixed RNNs can achieve arbitrary accuracy via runtime.
• Algorithmic freedom can be integrated into RNNs.
• Runtime is an unavoidable resource for fixed-network approximation.

Method

The construction uses a Turing machine with neural units (TMNU) to implement polynomial approximation schemes, which is then simulated by a ReLU RNN with explicit bounds on hidden dimension and weight magnitude.

Topics

• Recurrent Neural Networks
• Function Approximation
• ReLU Networks
• Turing Machine with Neural Units
• Approximation Theory

Best for: AI Scientist, Research Scientist

Related on AIssential

• The Evolution of Language Modeling — Recurrent Neural Networks process variable-length sequential data using feedback loops and shared weights.
• Dense Neural Networks are not Universal Approximators — Dense neural networks are not universal approximators under practical weight and dimension constraints.
• Memory Caching: RNNs with Growing Memory — Memory Caching enhances RNNs by enabling memory growth with sequence length, bridging the performance gap with Transformers.
• ParaRNN: Large-Scale Nonlinear RNNs, Trainable in Parallel — ParaRNN enables parallel training of large-scale nonlinear RNNs, matching transformer performance with superior inference efficiency.
• Improved state mixing in higher-order and block diagonal linear recurrent networks — Richer state mixing in LRNNs enhances expressivity and efficiency for long-sequence modeling.
• Weighted universal approximation of differentiable maps on infinite-dimensional manifolds — The universal approximation theorem for FNNs is generalized to differentiable maps, including derivative approximation, via a weighted Nachbin theorem.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.