One Hidden Layer Can Fit Any Curve

2026-06-07 · Source: DataMListic · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Novice, quick

Summary

A neural network featuring a single hidden layer possesses the geometric capability to approximate any arbitrary curve with high precision. This phenomenon stems from the behavior of individual neurons: when a neuron's weight is significantly increased, its sigmoid activation function sharpens into an approximate step function. By subtracting two such step functions oriented in opposite directions, a localized "bump" can be generated, which is adjustable in position and height. The Universal Approximation Theorem demonstrates that by combining a sufficient number of these customizable bumps, a single hidden layer network can effectively sum them to precisely trace and replicate complex curves, achieving any desired level of approximation accuracy.

Key takeaway

For machine learning engineers designing or debugging neural networks, understanding the Universal Approximation Theorem's geometric basis is crucial. Your single hidden layer network, even if simple, theoretically possesses the power to model highly complex relationships. Focus on sufficient neuron count and appropriate activation functions to ensure your models can capture intricate data patterns, rather than immediately assuming deeper architectures are always necessary for complex curve fitting.

Key insights

A single hidden layer neural network can approximate any curve by combining adjustable "bumps" derived from sigmoid step functions.

Principles

• High neuron weights transform sigmoids into step functions.
• Subtracting two step functions creates a localized "bump."
• Combining many such bumps approximates any continuous function.

Method

Construct a single hidden layer network where neurons generate step-like functions. Combine these to form adjustable bumps, then sum multiple bumps to approximate target curves.

Topics

• Neural Networks
• Universal Approximation Theorem
• Hidden Layers
• Sigmoid Activation
• Function Approximation
• Machine Learning Theory

Best for: AI Student, Machine Learning Engineer, Data Scientist

Related on AIssential

• Universality of Shallow and Deep Neural Networks on Non-Euclidean Spaces — TFNNs extend universal approximation to non-Euclidean spaces, even with width constraints, by leveraging topological properties.
• Activation Functions in Neural Networks - Explained — Activation functions introduce non-linearity, enabling neural networks to learn complex decision boundaries.
• Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approcimations — Neural flow operators provide a unified framework proving universal approximation for both finite and infinite-dimensional spaces.
• Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks — Curvature-aware optimization using secant information significantly improves PINN training stability and accuracy.
• Weighted universal approximation of differentiable maps on infinite-dimensional manifolds — The universal approximation theorem for FNNs is generalized to differentiable maps, including derivative approximation, via a weighted Nachbin theorem.
• Dense Neural Networks are not Universal Approximators — Dense neural networks are not universal approximators under practical weight and dimension constraints.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by DataMListic.