StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning

2026-06-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

StepPO is introduced as a step-aligned policy optimization framework for Agentic Reinforcement Learning (RL), addressing the limitations of token-centric modeling in multi-turn interactive settings. It advocates for advancing the conventional token-level Markov Decision Process (MDP) to a step-level MDP, where a complete interaction round, rather than individual tokens, serves as the action representation for Large Language Model (LLM) agents. StepPO proposes step-level credit assignment to align policy optimization and reward propagation with this decision granularity. The framework also outlines necessary system designs, including step-native data representation, gateway-centered data management, computational efficiency through shared-prefix reuse, and asynchronous training. Preliminary experiments on HotpotQA, using Qwen2.5-3B-Instruct, demonstrate that StepPO consistently outperforms token-level PPO in multi-step agent tasks, achieving superior performance throughout training.

Key takeaway

For Machine Learning Engineers developing LLM agents for multi-turn interactive tasks, you should transition from token-level to step-level policy optimization. Your current token-centric Reinforcement Learning (RL) methods may be inadequate for capturing complex agent behavior and delayed rewards. Adopting a step-level Markov Decision Process (MDP) and credit assignment, as demonstrated by StepPO, will provide a more effective learning signal, leading to stronger agent capabilities and improved performance in long-horizon tasks. Consider restructuring your training pipelines to support step-native data and asynchronous execution.

Key insights

Agentic RL requires aligning MDPs, credit assignment, and systems to the interaction step, not individual tokens.

Principles

• Interaction steps, not tokens, define agent actions.
• Align credit assignment with interaction step granularity.
• Step-native data and asynchronous systems are crucial.

Method

StepPO reformulates MDPs to step-level, applies step-level GAE for credit assignment, and uses structured step-native trajectory representation within asynchronous training systems.

In practice

• Evaluate step-level PPO for multi-turn agent tasks.
• Adopt step-native data structures for agent trajectories.
• Consider asynchronous training for agent RL systems.

Topics

• Agentic Reinforcement Learning
• Large Language Models
• Step-level MDP
• Policy Optimization
• Credit Assignment
• RL Training Systems

Code references

• AgentR1/Agent-R1
• AgentR1/Claw-R1

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories — Agentic-DPO converts expert trajectories into state-level action preferences for efficient, offline LLM agent policy optimization, surpassing imitation learning.
• APPO: Agentic Procedural Policy Optimization — APPO refines agentic RL credit assignment by shifting branching to fine-grained decision points, improving multi-turn tool-use.
• Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents — STEP-HRL uses hierarchical and local progress modules to enable efficient step-level learning for LLM agents.
• Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents — Semantic Consistency Policy Optimization (SCPO) resolves conflicting gradients in LLM agent reinforcement learning by assigning credit from successful sibling trajectories.
• Multi-Agent Reinforcement Learning for Dynamic Pricing: Balancing Profitability,Stability and Fairness — MARL methods, especially MAPPO, offer stable and scalable dynamic pricing solutions in competitive retail.
• From GRPO to SAMPO: Solving Training Collapse in Agentic RL — SAMPO stabilizes agentic RL training by optimizing four key policy dimensions to prevent catastrophic gradient issues.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.