Any-Time Regret-Guaranteed Algorithm for Control of Linear Quadratic Systems

2026-07-23 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

A new algorithm, ASLO, is introduced for Linear Quadratic (LQ) control systems with unknown models, addressing limitations of prior work like Cohen et al. (2019). ASLO is the first fully adaptive algorithm that dynamically adjusts the exploration-exploitation trade-off and optimizes regret. It builds on an SDP-based approach, relaxing the need for a horizon-dependent warm-up phase by tuning the regularization parameter (λt = O(log t)) and adding adaptive input perturbation (variance O(1/sqrt(t))). Crucially, ASLO eliminates the restrictive requirement for strong sequential stability by adaptively controlling policy updates. The algorithm achieves an overall regret of O(sqrt(T)), with its adaptive policy update mechanism improving certain regret terms from O(sqrt(T)) to O(T^(1/4)), demonstrating superior performance.

Key takeaway

For AI Scientists and Research Scientists developing adaptive control systems, this work demonstrates that dynamically adjusting exploration-exploitation trade-offs and relaxing stringent stability requirements can significantly improve performance. You should consider implementing adaptive regularization and input perturbation strategies, such as O(log t) and O(1/sqrt(t)) variance respectively, to achieve O(sqrt(T)) regret with faster convergence for certain terms. This approach reduces reliance on user-defined parameters and simplifies warm-up phases, leading to more robust and efficient control designs.

Key insights

Adaptive exploration-exploitation and relaxed stability criteria enable efficient, regret-guaranteed control for unknown linear quadratic systems.

Principles

• Dynamic adjustment of exploration-exploitation is crucial for optimal regret in adaptive control.
• Overly stringent stability conditions can introduce inefficiencies and extend warm-up phases.
• Adaptive input perturbation and regularization can replace horizon-dependent warm-up procedures.

Method

The ASLO algorithm employs a finite, horizon-independent warm-up phase for initial parameter estimation, followed by an adaptive SDP-based control that dynamically tunes regularization (λt = O(log t)) and input perturbation (variance O(1/sqrt(t))) to balance exploration and exploitation.

In practice

• Implement dynamic regularization (e.g., O(log t)) and input perturbation (e.g., O(1/sqrt(t)) variance) in adaptive control systems.
• Re-evaluate the necessity of strong sequential stability in control design to simplify initialization.
• Optimize policy update frequency using a monotonically decreasing function for exploration-exploitation trade-off.

Topics

• Linear Quadratic Control
• Adaptive Control
• Reinforcement Learning
• Regret Bounds
• Exploration-Exploitation
• Semi-Definite Programming

Best for: AI Scientist, Research Scientist

Related on AIssential

• Robust Exploratory Stopping under Ambiguity in Reinforcement Learning — A robust RL framework for optimal stopping under ambiguity balances exploration and exploitation using $g$-expectation and BSDEs.
• Wind-Aware Reinforcement Learning Control of a Small Quadrotor Using Learned Onboard Wind Estimation in Simulated Atmospheric Turbulence — A two-stage learning pipeline combining onboard wind estimation with RL control significantly improves quadrotor performance in turbulent atmospheric conditions.
• Regret Minimization with Adaptive Opponents in Repeated Games — Repeated Policy Regret (RP-Regret) enables finding higher-utility equilibria in repeated games by modeling adaptive opponents and dynamic strategies.
• Adaptive Multi-Horizon Reinforcement Learning — Adaptive multi-horizon reinforcement learning dynamically balances short-term and long-term consequences without manual discount-factor tuning.
• Catching a Moving Subspace: Low-Rank Bandits Beyond Stationarity — Scalar rewards enable quadratic measurements for changing low-rank subspace recovery in non-stationary bandits.
• Exploration and Exploitation: The Simple Yet Profound Logic at the Heart of Reinforcement Learning — Balancing exploration of new options with exploitation of known successes is central to learning under uncertainty.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.