Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Safe Flow Q-Learning (SafeFQL) is a novel framework for safe offline reinforcement learning, extending Flow Q-Learning to incorporate strict safety constraints. It combines a Hamilton-Jacobi reachability-inspired safety value function with an efficient one-step flow policy, avoiding iterative denoising or rejection sampling at deployment. SafeFQL includes a conformal prediction calibration step to adjust safety thresholds for finite-data learning errors, providing probabilistic safety coverage. Empirically, SafeFQL demonstrates modestly higher offline training costs but substantially lower inference latency compared to diffusion-style safe generative baselines, making it suitable for real-time safety-critical control. It matches or exceeds prior offline safe RL performance across boat navigation and Safety Gymnasium MuJoCo tasks, significantly reducing constraint violations.

Key takeaway

For Robotics Engineers or ML teams developing real-time safety-critical control systems from offline data, SafeFQL offers a compelling solution. Its one-step policy extraction significantly reduces inference latency compared to generative baselines, enabling deployment in high-frequency loops. You should consider SafeFQL to achieve robust, probabilistically guaranteed safety with high reward performance, especially where even single constraint violations are unacceptable. This approach minimizes the need for manual safety tuning and complex rejection sampling.

Key insights

SafeFQL ensures real-time, probabilistic safety in offline RL via a one-step flow policy and reachability-based value function.

Principles

• Prioritize safety over reward with a feasibility-gated objective.
• Use reachability-inspired max-backup for worst-case safety.
• Calibrate safety thresholds with conformal prediction for finite-sample guarantees.

Method

SafeFQL learns reward/safety critics, fits a behavior flow teacher, distills it into a one-step actor, and optimizes the actor with a feasibility-gated objective, followed by conformal calibration.

In practice

• Deploy SafeFQL for real-time safety-critical control systems.
• Apply to high-dimensional robotics tasks like MuJoCo environments.
• Use for collision avoidance in navigation systems.

Topics

• Safe Reinforcement Learning
• Offline RL
• Flow Q-Learning
• Hamilton-Jacobi Reachability
• Conformal Prediction
• Real-time Control
• MuJoCo

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Robust Shielding for Safe Reinforcement Learning — A new shielding framework for Robust MDPs guarantees safe reinforcement learning without requiring prior knowledge of transition dynamics.
• Drift Q-Learning — DriftQL combines drift-based regularization with critic-driven policy improvement for efficient, robust offline RL.
• Conformal Policy Control — Conformal Policy Control enables safe AI exploration by calibrating new policies against a safe baseline, provably controlling risk.
• Robust Peak-cost Constrained Reinforcement Learning — RP-CRL addresses safety-critical RL by controlling peak costs per trajectory and handling real-world dynamics mismatch, unlike standard CMDPs.
• Safe-RULE: Safe Reinforcement UnLEarning — Safe-RULE defends offline Safe RL from data poisoning by unlearning malicious data while preserving safety and task performance.
• RL-STPA: Adapting System-Theoretic Hazard Analysis for Safety-Critical Reinforcement Learning — RL-STPA systematically identifies and mitigates hazards in safety-critical RL systems through adapted STPA principles.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.