The Anatomy of an Optimization Problem

2026-07-10 · Source: Steve Brunton · Field: Science & Research — Mathematics & Computational Sciences, Engineering & Applied Sciences · Depth: Intermediate, extended

Summary

An optimization problem fundamentally comprises an objective function f(x) and a set of constraint equations. The objective function, which can be minimized or maximized, takes variables x (potentially high-dimensional, like neural network weights) and yields a scalar output. It can be convex, like in least squares regression, possessing a single global minimum, or non-convex, as seen in neural network training, featuring multiple local minima and saddle points. The gradient of f is crucial for iterative minimization algorithms. Constraint equations define a feasible region, which similarly can be convex (e.g., linear inequalities forming convex polytopes) or non-convex (e.g., non-linear constraints). Challenges in solving these problems include non-convexity, expensive objective function evaluations (e.g., \$10,000 for a super alloy test), high-dimensional variables, ill-conditioning, and absent gradient information. Key problem classes include linear programming and quadratic programming, both often convex.

Key takeaway

For Machine Learning Engineers or Data Scientists evaluating model training strategies, understanding the convexity of your objective function and feasible set is critical. Non-convex problems, common in deep learning, necessitate iterative or stochastic solutions, often without global optimality guarantees. Conversely, convex problems like least squares are largely solved with scalable algorithms. Consider the computational cost of objective function evaluation and variable dimensionality when selecting optimization techniques.

Key insights

The core of optimization involves an objective function and constraints, whose properties dictate solution complexity.

Principles

• Convex objective functions or feasible sets simplify optimization.
• Non-convexity in either objective or constraints complicates global minimum discovery.

Method

Iterative gradient descent algorithms use the negative gradient direction to find local minima, especially for high-dimensional, non-convex problems like neural network training.

In practice

• Use gradient descent for neural network parameter optimization.
• Recognize linear programming for finance and logistics scheduling.

Topics

• Optimization Theory
• Objective Functions
• Constraint Equations
• Convexity
• Gradient Descent
• Linear Programming
• Neural Networks

Best for: AI Student, Machine Learning Engineer, Data Scientist

Related on AIssential

• Convexity 101 [Optimization Bootcamp] — If an optimization problem is convex, any local minimum is guaranteed to be a global minimum, simplifying solution finding.
• Revisiting a crazy global NLP problem — Non-convex quadratic optimization problems can be intractable even for small instances.
• Optimization: A Bootcamp for Machine Learning, Inverse Problems, and Control — Optimization is the ubiquitous mathematical engine underpinning machine learning, control theory, and industrial applications, solving problems by minimizing objectives under constraints.
• Subject to: Aharon Ben-Tal — Ronnie Ben-Tal's career exemplifies the profound impact of theoretical optimization on real-world engineering and medical challenges.
• Advancing Optimal Subset Oracle via Learning Relaxation of Neural Set Functions — Replacing sampling-based ELBO gradient estimation with a learned surrogate objective stabilizes and accelerates neural set function optimization.
• Decentralized Stochastic Subgradient-type Methods with Communication Compression for Nonsmooth Nonconvex Optimization — A unified framework ensures global convergence for decentralized nonsmooth nonconvex optimization with communication compression.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Steve Brunton.