Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

Researchers investigated the use of symbolic regression to discover explicit neural network weight-update rules, aiming to outperform standard hand-designed optimizers on small symbolic regression benchmarks. Candidate update rules were represented as fixed-depth symbolic expressions, utilizing operands derived from common optimizers such as gradient, momentum, adaptive-gradient, and moment-estimate quantities. Across 30 benchmark and neural network combinations, the symbolic regression procedure successfully identified an update rule that surpassed the best hyperparameter-tuned established optimizer in 25 cases. This resulted in an aggregate Mean Squared Error (MSE) reduction of 44.47% over the improved instances. The discovered rules, while diverse, often combined adaptive normalization, momentum-like quantities, nonlinear transformations, and rational expressions, indicating symbolic regression's potential for finding compact optimizer variants, pending larger-scale validation.

Key takeaway

For Machine Learning Engineers optimizing neural network performance, consider exploring symbolic regression to discover custom weight-update rules. This approach has demonstrated significant MSE reductions, outperforming established optimizers in many cases. You should investigate integrating adaptive normalization, momentum-like quantities, and nonlinear transformations into your optimizer design. While promising, remember this method requires larger-scale validation for broader applicability beyond small benchmarks.

Key insights

Symbolic regression can discover neural network weight-update rules that outperform hand-designed optimizers on benchmarks.

Principles

• Optimizer rules can be learned via symbolic regression.
• Combining adaptive elements improves update rules.
• Fixed-depth symbolic expressions are effective representations.

Method

Symbolic regression searches fixed-depth expressions using operands from gradient, momentum, adaptive-gradient, and moment-estimate quantities to find superior weight-update rules.

In practice

• Explore symbolic regression for custom optimizers.
• Integrate adaptive normalization in rule design.
• Test nonlinear transformations in update functions.

Topics

• Symbolic Regression
• Neural Network Optimizers
• Weight Update Rules
• Adaptive Normalization
• Momentum
• Machine Learning Benchmarks

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics — Gradient descent in ReLU networks can be modeled as collective dynamics in training-set space, mediated by layer-wise Gram operators.
• FunctionEvolve: Structure-Guided Symbolic Regression with LLMs — Structure-guided symbolic regression with LLMs significantly improves exact form recovery by using expression trees.
• Training Neural Networks with Optimal Double-Bayesian Learning — A double-Bayesian framework theoretically derives optimal SGD hyperparameters, showing superior generalization over Adam.
• Weight Adaptation for Improving Parallel Performance of Adaptive Stochastic Natural Gradient — Adaptively adjusting weight parameters in stochastic natural gradient methods maximizes optimization improvement, especially for parallel black-box problems.
• Scaling Neuro-symbolic Problem Solving: Solver-Free Learning of Constraints and Objectives — A neuro-symbolic architecture with a probabilistic loss efficiently learns to solve NP-hard problems from natural inputs.
• Why Gradient Descent Became Stochastic — Gradient Descent and its variants offer scalable alternatives to the computationally intensive Normal Equation for optimizing linear regression on large datasets.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.