Functional Gradient Descent with Adaptive Representations

2026-05-07 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Engineering & Applied Sciences · Depth: Expert, extended

Summary

The paper introduces a novel, theoretically-grounded Functional Gradient Descent (FGD) algorithm that adaptively refines the representation of functional gradients during optimization. Traditional FGD implementations struggle with infinite-dimensional functional gradients, relying on fixed approximations that introduce error. This new method establishes convergence to a stationary point for smooth losses and a global minimizer under a Polyak-Łojasiewicz-type condition, regardless of approximation. It achieves this by adaptively refining gradient approximations to maintain a specific relative error bound. Demonstrated on regression in an RKHS, numerical PDE solutions (wave equation), and modern computer vision (radiance fields), the method consistently outperforms both fixed-approximation FGD and neural network baselines in efficiency and accuracy. Experiments were run on an Intel Core i9-14900KF CPU and NVIDIA GeForce RTX 4090 GPU with 128GB RAM and 24GB VRAM.

Key takeaway

For Machine Learning Engineers and Research Scientists tackling functional optimization problems, this adaptive FGD method offers a robust alternative to traditional neural network training or fixed-approximation FGD. You should consider implementing this approach, especially for tasks like PDE solving or inverse rendering, as it promises faster convergence to global minimizers and superior solution quality, reducing reliance on highly nonconvex neural network losses.

Key insights

Adaptive representation in functional gradient descent ensures convergence to global minimizers by dynamically refining gradient approximations.

Principles

• Maintain a relative error bound for gradient approximations.
• Convergence guaranteed for smooth losses and Polyak-Łojasiewicz conditions.
• Functional gradients can be approximated in broader Banach spaces.

Method

The algorithm iteratively refines gradient approximations by designing a sequence of representations and selecting one that satisfies a computable relative error bound.

In practice

• Apply to regression tasks in RKHSs.
• Solve partial differential equations (PDEs).
• Optimize radiance fields for inverse rendering.

Topics

• Functional Gradient Descent
• Adaptive Optimization
• Functional Optimization
• Partial Differential Equations
• Neural Radiance Fields
• Machine Learning Theory

Code references

• dccsillag/experiments-adaptive-fgd

Best for: Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents — This paper proposes a novel Hybrid Selection Strategy (HSS) for Kernel-Based Gradient Descent (KGD) algorithms, which integrates bias-variance analysis with the splitting method for adaptive parameter selection.
• Learning functional components of PDEs from data using neural networks — Neural networks can approximate unknown functional components within PDEs by embedding and training them with data.
• Mitigating Forgetting in Continual Learning with Selective Gradient Projection — SFAO uses similarity-gated gradient updates to balance plasticity and stability in continual learning, reducing forgetting and memory.
• Attractor FCM — The Attractor FCM uses Jacobian gradient descent and adaptive learning to achieve robust, interpretable, and physics-informed simulations.
• Gradient Regularized Newton Boosting Trees with Global Convergence — A gradient-regularized Newton boosting scheme achieves global convergence for GBDTs, addressing prior theoretical gaps.
• Weighted universal approximation of differentiable maps on infinite-dimensional manifolds — The universal approximation theorem for FNNs is generalized to differentiable maps, including derivative approximation, via a weighted Nachbin theorem.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.