Safe Reinforcement Learning using Ideas from Model Predictive Control

2026-07-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, long

Summary

A generalized framework combines the adaptive, high-performance nature of deep reinforcement learning (DRL) with the formal safety guarantees of model predictive control (MPC). This approach addresses the critical challenge of ensuring strict safety constraints during RL's active learning phase, particularly for complex cyber-physical systems and robotics where unconstrained exploration risks irreversible damage. The framework utilizes offline MPC computations to pre-define a feasible state-action space, encompassing all safe system states and control inputs. During both training and deployment, the RL agent's instantaneous actions are projected onto this globally verified safe set via a deterministic safety filter. Empirical validation on a non-linear 1-DOF Quanser Aero 2 laboratory testbed demonstrated successful exploration and stable policy convergence on physical hardware, despite minor deviations attributed to the "Sim-to-Real" gap.

Key takeaway

For Machine Learning Engineers deploying reinforcement learning agents on physical hardware, this framework provides a critical solution for ensuring strict safety. You should integrate an offline Model Predictive Control-derived safety filter to project proposed actions into a pre-verified safe space. This approach prevents catastrophic exploration failures and stabilizes training, even with minor Sim-to-Real discrepancies, enabling safer and more reliable real-world RL applications.

Key insights

Combining offline Model Predictive Control with a real-time projection filter enables safe Reinforcement Learning on physical systems.

Principles

• RL exploration must respect strict physical safety constraints.
• Offline MPC can pre-define recursively safe state-action spaces.
• System convexity allows efficient mapping of safe action boundaries.

Method

Offline MPC computes a feasible state-action space. During RL training, a projection filter maps the agent's proposed actions to the nearest safe point within this pre-computed space, ensuring recursive safety.

In practice

• Apply to non-linear cyber-physical systems and robotics.
• Implement a projection filter for real-time action safety.
• Penalize projection magnitude in RL reward for native safety.

Topics

• Safe Reinforcement Learning
• Model Predictive Control
• Cyber-Physical Systems
• Robotics Control
• Safety Filters
• Quanser Aero 2

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Safe Reinforcement Learning using Ideas from Model Predictive Control — Integrating MPC's formal safety guarantees with DRL's adaptability ensures strict constraint satisfaction in real-world RL systems.
• Learning of Robot Safety Policies via Adversarial Synthetic Scenarios — An adversarial gamification framework enables robust robot safety policy learning by iteratively discovering high-risk edge cases.
• Beyond Conservative Automated Driving in Multi-Agent Scenarios via Coupled Model Predictive Control and Deep Reinforcement Learning — Coupling RL for speed guidance with MPC for trajectory optimization improves autonomous driving safety and efficiency.
• Robust Shielding for Safe Reinforcement Learning — A new shielding framework for Robust MDPs guarantees safe reinforcement learning without requiring prior knowledge of transition dynamics.
• Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies — SafeFQL ensures real-time, probabilistic safety in offline RL via a one-step flow policy and reachability-based value function.
• Safe-RULE: Safe Reinforcement UnLEarning — Safe-RULE defends offline Safe RL from data poisoning by unlearning malicious data while preserving safety and task performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.