A note on convergence of Wasserstein policy optimization

2026-05-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Optimization and Control · Depth: Expert, quick

Summary

Wasserstein Policy Optimization (WPO), a recently proposed reinforcement learning algorithm designed for stochastic policies in continuous action spaces, leverages Wasserstein gradient flows for optimization. Despite its empirical performance, WPO's theoretical convergence in continuous state and action environments has been unconfirmed. This note posits that WPO, when applied within entropy-regularized Markov Decision Processes, achieves linear convergence. The argument relies on recent advancements in mean-field analysis for gradient flow convergence, specifically utilizing log-Sobolev inequalities. By assuming a sufficiently regular solution to the gradient flow equation, the authors demonstrate monotonic energy dissipation along the flow and establish a local log-Sobolev inequality. These findings collectively support the claim that the value function converges linearly to the global optimum.

Key takeaway

For AI Scientists and RL researchers evaluating policy optimization algorithms, this work provides crucial theoretical validation for Wasserstein Policy Optimization (WPO). You can now consider WPO with greater confidence, knowing its linear convergence in entropy-regularized Markov Decision Processes is formally established. This understanding supports its application in continuous action spaces and informs future algorithm design by highlighting the role of mean-field analysis and log-Sobolev inequalities in convergence proofs.

Key insights

Wasserstein Policy Optimization (WPO) converges linearly in entropy-regularized Markov Decision Processes, leveraging log-Sobolev inequalities.

Principles

• Wasserstein gradient flows optimize stochastic policies.
• Entropy regularization enables linear convergence proof.
• Log-Sobolev inequalities establish convergence rates.

Method

Prove WPO linear convergence by applying mean-field analysis and log-Sobolev inequalities to gradient flows. Assume regular solutions, then demonstrate monotonic energy dissipation and establish a local log-Sobolev inequality.

Topics

• Wasserstein Policy Optimization
• Reinforcement Learning
• Gradient Flows
• Markov Decision Processes
• Log-Sobolev Inequalities
• Convergence Theory

Best for: Research Scientist, AI Scientist

Related on AIssential

• Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting — PG-DPO offers a non-recursive, Pontryagin-guided variational framework for reinforcement learning with non-exponential discounting, improving stability where Bellman recursions fail.
• Optimal last-iterate convergence in matrix games with bandit feedback using the log-barrier — Log-barrier regularization enables optimal last-iterate convergence in uncoupled zero-sum matrix games.
• Model-Driven Policy Optimization in Differentiable Simulators via Stochastic Exploration — MDPO enhances differentiable planning by adaptively injecting stochastic noise for improved exploration in complex optimization landscapes.
• A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization — RLVR optimization stability hinges on advantage sign and token distribution, addressed by WAPO's positive-advantage updates.
• Entropy-Preserving Reinforcement Learning — Policy gradient algorithms often reduce exploration diversity; active entropy control improves performance and trainability.
• MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following — MDP-GRPO stabilizes GRPO for multi-constraint instruction following by addressing z-score normalization pathologies with specific algorithmic enhancements.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.