Optimization: A Bootcamp for Machine Learning, Inverse Problems, and Control

2026-07-02 · Source: Steve Brunton · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Mathematics & Computational Sciences · Depth: Intermediate, long

Summary

Steve Brunton from the University of Washington is launching an "Optimization Bootcamp," a comprehensive video series designed to explain the mathematical engine behind modern technology and industry. This bootcamp will cover the standard optimization problem, which involves minimizing an objective function f over variables x subject to constraints g and h. It will explore diverse applications, including the training of virtually all machine learning models (e.g., neural networks via stochastic gradient descent), control theory, and complex industrial design challenges like the Boeing 787 aircraft and chip manufacturing. The series will delve into mathematical methods, distinguishing between convex and non-convex optimization, and discussing techniques such as gradient descent, linear programming, and quadratic programming. Future modules will address inverse problems, Bayesian optimization, and the role of Lagrange multipliers in handling constraints.

Key takeaway

For Machine Learning Engineers or Research Scientists seeking to deepen their understanding of model training and system optimization, this bootcamp offers foundational knowledge. You will learn how optimization underpins virtually all ML models and control systems, enabling you to better diagnose training issues or design more efficient algorithms. Consider exploring convex optimization principles to identify problems with guaranteed global minima, enhancing your ability to select appropriate solution methods.

Key insights

Optimization is the ubiquitous mathematical engine underpinning machine learning, control theory, and industrial applications, solving problems by minimizing objectives under constraints.

Principles

• Optimization problems minimize an objective function f(x) subject to constraints g(x) and h(x).
• Convex optimization guarantees a local minimum is also a global minimum.
• Gradient descent iteratively finds local minima by moving in the negative gradient direction.

Method

The bootcamp outlines a method to solve optimization problems by defining objective functions and constraints, then applying techniques like gradient descent or specialized convex optimization algorithms.

In practice

• Train machine learning models using stochastic gradient descent.
• Optimize supply chains and factory layouts with linear programming.
• Solve least squares regression problems via quadratic programming.

Topics

• Optimization Theory
• Machine Learning Training
• Control Theory
• Convex Optimization
• Gradient Descent
• Linear Programming
• Inverse Problems

Best for: AI Student, Machine Learning Engineer, Research Scientist

Related on AIssential

• stochastic gradient descent #maths #statistics #datascience #machinelearning #dataanlysis — Faster iteration with noisy gradients often outperforms slow, precise convergence in machine learning.
• MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resources — MiniOpt is an RL framework that enables compact LLMs to generalize across diverse optimization problems using a "reasoning-to-model-and-solve" paradigm.
• Subject to: Aharon Ben-Tal — Ronnie Ben-Tal's career exemplifies the profound impact of theoretical optimization on real-world engineering and medical challenges.
• AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm — AdaGrad-Diff improves optimization robustness by adapting stepsizes based on gradient differences, not just gradient magnitudes.
• LEAF: A Learning-Enabled ADMM Framework for Accelerated Convex Optimization — LEAF uses ICNNs to learn a scalar Moreau envelope, accelerating convex optimization with reduced complexity.
• Gradient Descent — An Explanation — Gradient Descent optimizes functions by iteratively moving towards a local minimum using the function's gradient.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Steve Brunton.