StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning

2026-06-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

StepPO is introduced as a step-aligned policy optimization framework for Agentic Reinforcement Learning (RL), addressing the limitations of token-centric modeling in multi-turn interactive settings. It advocates for advancing the conventional token-level Markov Decision Process (MDP) to a step-level MDP, where a complete interaction round, rather than individual tokens, serves as the action representation for Large Language Model (LLM) agents. StepPO proposes step-level credit assignment to align policy optimization and reward propagation with this decision granularity. The framework also outlines necessary system designs, including step-native data representation, gateway-centered data management, computational efficiency through shared-prefix reuse, and asynchronous training. Preliminary experiments on HotpotQA, using Qwen2.5-3B-Instruct, demonstrate that StepPO consistently outperforms token-level PPO in multi-step agent tasks, achieving superior performance throughout training.

Key takeaway

For Machine Learning Engineers developing LLM agents for multi-turn interactive tasks, you should transition from token-level to step-level policy optimization. Your current token-centric Reinforcement Learning (RL) methods may be inadequate for capturing complex agent behavior and delayed rewards. Adopting a step-level Markov Decision Process (MDP) and credit assignment, as demonstrated by StepPO, will provide a more effective learning signal, leading to stronger agent capabilities and improved performance in long-horizon tasks. Consider restructuring your training pipelines to support step-native data and asynchronous execution.

Key insights

Agentic RL requires aligning MDPs, credit assignment, and systems to the interaction step, not individual tokens.

Principles

• Interaction steps, not tokens, define agent actions.
• Align credit assignment with interaction step granularity.
• Step-native data and asynchronous systems are crucial.

Method

StepPO reformulates MDPs to step-level, applies step-level GAE for credit assignment, and uses structured step-native trajectory representation within asynchronous training systems.

In practice

• Evaluate step-level PPO for multi-turn agent tasks.
• Adopt step-native data structures for agent trajectories.
• Consider asynchronous training for agent RL systems.

Topics

• Agentic Reinforcement Learning
• Large Language Models
• Step-level MDP
• Policy Optimization
• Credit Assignment
• RL Training Systems

Code references

• AgentR1/Agent-R1
• AgentR1/Claw-R1

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents — STEP-HRL uses hierarchical and local progress modules to enable efficient step-level learning for LLM agents.
• MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following — MDP-GRPO stabilizes GRPO for multi-constraint instruction following by addressing z-score normalization pathologies with specific algorithmic enhancements.
• Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling — GDCR and SAPO provide step-level credit assignment for agentic search by leveraging graph-based distance to an answer node.
• Structured Role-Aware Policy Optimization for Multimodal Reasoning — Role-aware token-level credit assignment significantly improves multimodal reasoning in LVLMs.
• From GRPO to SAMPO: Solving Training Collapse in Agentic RL — SAMPO stabilizes agentic RL training by optimizing four key policy dimensions to prevent catastrophic gradient issues.
• Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards — Augmenting PPO with multi-step likelihood ratios provides a principled way to control bias-variance in RLVR.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.