A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function

2026-07-04 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

Roberto Rossi's work offers a novel interpretation of the Gaussian Error Linear Unit (GELU) activation function, traditionally understood as an input-dependent Bernoulli gate. This research redefines GELU as the expected output of a hard linear gate with a Gaussian random threshold, providing a generative view where the gate activates upon clearing a latent Gaussian threshold. This interpretation, derived from stochastic inventory theory, introduces a "threshold-transmission family" encompassing established functions like ReLU, GELU, SiLU/Swish, and hard swish. Furthermore, by employing a latent uniform threshold, the study develops a hard-swish-like piecewise-polynomial gate with a confined nonlinear transition interval, available in fixed- and learned-width variants. Controlled experiments on compact vision and language models demonstrate that these calibrated or learned uniform-threshold gates are consistently competitive with, and often outperform, GELU, ReLU, and SiLU/Swish, utilizing their finite transition regions effectively.

Key takeaway

For Machine Learning Engineers optimizing model performance, this research suggests exploring the new uniform-threshold activation functions. If you are currently using ReLU, GELU, or SiLU/Swish, consider implementing calibrated or learned uniform-threshold gates, as experiments show they are often competitive or superior in compact vision and language models. This could lead to improved model efficiency and accuracy in your deployments.

Key insights

The paper reinterprets GELU via a Gaussian random threshold, unifying common activation functions into a "threshold-transmission family".

Principles

• GELU can be viewed as a hard linear gate with a Gaussian random threshold.
• Stochastic inventory theory can unify activation functions.
• Uniform-threshold gates offer competitive performance.

Method

The work decomposes GELU based on stochastic inventory theory to derive a threshold-transmission family. It then applies latent uniform thresholds to create piecewise-polynomial gates.

In practice

• Implement calibrated uniform-threshold gates.
• Test new gates in compact vision and language models.

Topics

• GELU Activation Function
• Activation Functions
• Threshold-Transmission Gates
• Stochastic Inventory Theory
• Vision Models
• Language Models

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Activation Functions: The Hidden Switch Behind Every Neural Network — Activation functions are crucial for neural networks to learn complex, non-linear patterns, transforming simple linear operations into capabilities for real-world tasks.
• Encoding orders and trees in real-valued functions — The paper improves bounds for extracting order properties and thresholds from 2-trees using function-theoretic analogues.
• Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models — Improved threshold calibration and dynamic token routing enhance LLM inference efficiency and quality-throughput trade-offs.
• Fast approximation and learning of binary classification tasks in o-minimal structures using ReLU neural networks — ReLU neural networks can approximate complex decision regions in o-minimal structures, yielding quantitative approximation and statistical learning rates.
• Gate-Zero Growth: A Geometric Framework for Function-Preserving Continual Learning — Function-preserving continual learning is achieved by adding zero-initialized gated residual blocks that control functional variation.
• Information Theory and Statistical Learning — Information theory's divergence measures are fundamental for understanding and training diverse statistical learning models.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.