Retry Policy Gradients in Continuous Action Spaces

2026-06-06 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Researchers Soichiro Nishimori and Paavo Parmas from The University of Tokyo introduce Retry Policy Gradients for continuous action spaces, extending retry-based objectives like ReMax from discrete settings. Their work, ReMax Actor-Critic (ReMAC), is an off-policy actor-critic algorithm that uses a pathwise derivative estimator to optimize the ReMax objective. ReMAC promotes stochastic exploration without explicit entropy regularization by reshaping the policy-gradient landscape, biasing updates towards higher policy entropy and damping gradients to slow convergence. They demonstrate that Adam's adaptive normalization can mitigate this damping effect. Empirical evaluations across six Brax continuous-control tasks, including Ant and HalfCheetah, show ReMAC with retry budgets M>1 achieves performance comparable to Soft Actor-Critic (SAC) and consistently yields higher policy entropy than M=1 settings, particularly for M=4 and M=8 with Adam's default ε=10⁻⁸.

Key takeaway

For Machine Learning Engineers developing continuous control agents, consider ReMax Actor-Critic (ReMAC) as an alternative to entropy-regularized methods like SAC. ReMAC can achieve comparable performance while naturally encouraging exploration and higher policy entropy with retry budgets M>1. You should experiment with Adam's ε parameter to fine-tune the balance between exploration and convergence speed, potentially increasing it to restore ReMax's damping effect.

Key insights

ReMax extends to continuous action spaces, promoting exploration by reshaping policy gradients without entropy bonuses.

Principles

• Retry objectives promote exploration by adapting to return uncertainty.
• ReMax alters gradients to increase policy entropy and slow convergence.
• Adam's adaptive normalization can counteract ReMax's gradient damping.

Method

ReMAC is an off-policy actor-critic algorithm. It uses a pathwise derivative estimator to optimize the ReMax objective, replacing SAC's entropy bonus with the ReMax loss.

In practice

• Implement ReMAC by modifying SAC's actor loss.
• Adjust Adam's ε to control ReMax's gradient damping.
• Consider M>1 for higher policy entropy in continuous control.

Topics

• Continuous Control
• ReMax Objective
• Policy Gradients
• Actor-Critic Methods
• Exploration in RL
• Adam Optimizer

Code references

• nissymori/ReMAC
• keraJLi/rejax

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Retry Policy Gradients in Continuous Action Spaces — ReMax, extended to continuous action spaces, promotes stochastic exploration by reshaping policy gradients.
• Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying — ReMax induces emergent stochastic exploration in RL by optimizing for expected maximum return over retries, eliminating explicit bonuses.
• Reflex: Reinforcement Learning with Reflection Symmetry Exploitation in State-Based Continuous Control — Exploiting reflection symmetry in state-based RL significantly improves sample efficiency and performance.
• Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition — Second-order actor-critic methods can achieve stable, efficient updates by decomposing the policy Hessian.
• Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning — Adaptive entropy regularization dynamically balances exploration and exploitation in LLM reinforcement learning, preventing policy entropy collapse.
• Entropy-Preserving Reinforcement Learning — Policy gradient algorithms often reduce exploration diversity; active entropy control improves performance and trainability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.