A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

2025-12-31 · Source: JMLR · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A new stochastic proximal quasi-Newton method is proposed for minimizing the sum of two convex functions, specifically when one is an average of many smooth functions and the other is nonsmooth. Published in 27(103):1−43, 2026, this method integrates a single-loop SVRG (L-SVRG) technique for gradient sampling and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian. It demonstrates a globally linear convergence rate under mild assumptions and can generalize by incorporating other variance reduction methods like SAGA or SEGA. The approach includes an efficient, easily implementable semismooth Newton solver for nonsmooth subproblems, achieving arithmetic operations per iteration of O(d). Its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

Key takeaway

For research scientists optimizing large-scale nonsmooth convex functions, this new single-loop stochastic proximal quasi-Newton method offers a robust, efficient approach with guaranteed linear convergence. You should evaluate its O(d) semismooth Newton solver for problems like regularized logistic regression to enhance computational efficiency and convergence speed in your models. This method provides a strong foundation for integrating advanced variance reduction techniques.

Key insights

Integrating L-SVRG and stochastic L-BFGS yields a linearly convergent quasi-Newton method for nonsmooth convex optimization.

Principles

• Combine variance reduction with quasi-Newton methods for improved optimization.
• Efficient subproblem solvers are critical for practical algorithm performance.
• Globally linear convergence is achievable for complex convex problems.

Method

The method integrates single-loop SVRG for gradient sampling and stochastic L-BFGS for Hessian approximation, solving nonsmooth subproblems with a semismooth Newton solver.

In practice

• Apply to large-scale regularized logistic regression problems.
• Consider SAGA or SEGA as alternatives to L-SVRG for generalization.
• Utilize compact L-BFGS matrix representation for computational efficiency.

Topics

• Stochastic Optimization
• Quasi-Newton Methods
• Nonsmooth Convex Optimization
• SVRG
• L-BFGS
• Semismooth Newton Solver
• Logistic Regression

Best for: AI Scientist, Research Scientist

Related on AIssential

• Quantized Stochastic Primal-Dual Methods for Distributed Optimization under Relaxed Global Geometry — q-PDGD offers efficient distributed optimization with quantized communication, matching centralized stochastic rates under relaxed geometric conditions.
• Variance Reduction for Non-Log-Concave Sampling with Applications to Inverse Problems — Unified analysis shows variance reduction improves sampling from non-log-concave distributions, enhancing convergence and sample quality.
• Newton's Method is Just Tangent Lines — Newton's Method efficiently finds function roots and optimization minima by iteratively solving local linear or parabolic approximations.
• SPDE Methods for Nonparametric Bayesian Posterior Contraction and Laplace Approximation — Extending diffusion-based methods to SPDEs enables robust analysis of nonparametric Bayesian posteriors.
• Last-Iterate Convergence of Optimistic Multiplicative Weight Update — OMWU's last-iterate convergence for smooth convex-concave saddle-point problems is now theoretically proven, resolving a long-standing open question.
• Gradient Regularized Newton Boosting Trees with Global Convergence — A gradient-regularized Newton boosting scheme achieves global convergence for GBDTs, addressing prior theoretical gaps.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by JMLR.