Hybrid Least Squares/Gradient Descent Methods for MIONets

2026-07-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new hybrid least squares/gradient descent (LSGD) method is proposed for MIONets, aiming to significantly accelerate their training. This approach extends the LSGD method previously developed for DeepONets. MIONets are characterized as multilinear functions concerning the last layer parameters of each branch network, allowing for optimization via an alternating least squares (ALS) method. This involves sequentially solving a least squares system for individual branch networks. To manage large system matrices encountered in this process, the method incorporates Kronecker and Khatri-Rao products alongside tensor permutation matrices, enabling the factorization of large matrices into smaller, more manageable ones. The proposed LSGD method is compatible with a general type of L^2 loss, including regularization terms for the last layer parameters of each branch, and supports linear operators applied to the MIONet output within each loss term.

Key takeaway

For Machine Learning Engineers training MIONets, this hybrid least squares/gradient descent (LSGD) method offers a substantial acceleration by exploiting the network's multilinear properties. You should consider adapting this approach, especially when dealing with large-scale MIONet models, as it provides an efficient way to optimize last layer parameters and manage computational complexity through matrix factorization techniques.

Key insights

Efficient hybrid LSGD method accelerates MIONet training by leveraging its multilinear structure for alternating least squares optimization.

Principles

• MIONets are multilinear functions of last layer parameters.
• Alternating least squares optimizes multilinear neural network parameters.
• Tensor products factor large system matrices in LS problems.

Method

The LSGD method for MIONets optimizes last layer branch parameters using alternating least squares, solving a system for each branch sequentially. It employs Kronecker and Khatri-Rao products to factor large system matrices.

In practice

• Accelerate MIONet training workflows.
• Optimize multilinear neural network architectures.
• Handle large system matrices in least squares problems.

Topics

• MIONets
• DeepONets
• Least Squares
• Gradient Descent
• Alternating Least Squares
• Neural Network Training
• Tensor Products

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equation — The NPDG method solves PDEs via an inf-sup problem and preconditioned PDHG, yielding robust, accurate neural network training.
• Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models — Extended Random Matrix Theory unifies analysis of high-dimensional, nonlinear deep learning phenomena.
• Amortizing Maximum Inner Product Search with Learned Support Functions — Amortized MIPS uses neural networks to predict optimal keys by modeling the support function or directly regressing keys.
• Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents — This paper proposes a novel Hybrid Selection Strategy (HSS) for Kernel-Based Gradient Descent (KGD) algorithms, which integrates bias-variance analysis with the splitting method for adaptive parameter selection.
• Multi-ResNets for Subspace Preconditioning in Constrained Optimization — MResOpt uses staged residual networks and prioritized constraint decomposition for efficient constrained optimization.
• GPT-OSS 20B: A Sparse MoE Pretraining Benchmark for MLPerf Training v6.0 — GPT-OSS 20B offers a stable, accessible benchmark for sparse Mixture-of-Experts LLM pretraining in MLPerf Training v6.0.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.