Agentic RL: Environments, Trajectories, and the Training

2026-07-11 · Source: Daily Dose of Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, medium

Summary

This content details the process of assembling a full training loop for multi-step LLM agents, building on prior work that used an LLM-as-a-judge for relative scoring in single-response tasks. It addresses the challenge of generating rewards for non-verifiable, multi-step agent interactions where environments provide no explicit feedback. The article defines the agentic setting, mapping actions to model outputs or tool calls, observations to tool results, and state to the interaction history within a partially observable Markov decision process (POMDP). The core training unit becomes a complete "trajectory," encompassing the goal, all actions, observations, and the final answer. The objective is to maximize the expected reward over these entire trajectories, with the GRPO optimizer adapted to score these multi-step sequences. The OpenPipe ART framework and its RULER reward function are introduced for implementing this complete rollout, score, and update cycle.

Key takeaway

For AI Engineers developing multi-step LLM agents, understanding how to generate effective rewards is crucial. You should transition from single-response evaluation to scoring complete interaction trajectories, leveraging an LLM-as-a-judge for reliable relative comparisons. This approach, integrated with optimizers like GRPO, enables robust policy updates even in partially observable environments. Consider utilizing frameworks such as OpenPipe ART with its RULER reward function to streamline the implementation of this agentic RL training loop.

Key insights

Training multi-step LLM agents requires scoring full interaction trajectories using an LLM-as-a-judge within a GRPO framework.

Principles

• Relative scoring is more reliable than absolute scoring for LLM judges.
• Agent state is the interaction history in partially observable environments.
• Reward for multi-step agents is sparse and trajectory-level.

Method

Define agent actions, observations, and state as interaction history. Generate full trajectories, then use an LLM-as-a-judge for relative scoring. GRPO updates the policy based on these trajectory scores.

In practice

• Use OpenPipe ART framework for agent training.
• Implement RULER reward function for trajectory scoring.
• Map agent interactions to MDP concepts (actions, observations, state).

Topics

• Agentic Reinforcement Learning
• LLM Agents
• Trajectory Optimization
• LLM-as-a-Judge
• GRPO
• OpenPipe ART

Code references

• OpenPipe/ART

Best for: Machine Learning Engineer, AI Engineer, AI Scientist

Related on AIssential

• Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories — Agentic-DPO converts expert trajectories into state-level action preferences for efficient, offline LLM agent policy optimization, surpassing imitation learning.
• The Five Stages of Reinforcement Learning — Reinforcement learning for LLM agents involves five distinct stages, with reward design being a critical, nuanced component.
• Agent RL Training Frameworks: 10 Open-source Tools to Know — Open-source agent RL training frameworks offer diverse solutions for building and optimizing AI agents across various task complexities and deployment scales.
• GRPO Does Not Close the Multi-Agent Coordination Gap — Large language models struggle with multi-agent coordination, and GRPO does not improve it; training methodology is the core bottleneck.
• Agentic RL: Frameworks and Best Practices — Agentic RL requires specialized frameworks and techniques for stable, scalable multi-turn training with LLMs.
• Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization — CW-GRPO improves LLM search agents by using an LLM judge for fine-grained, per-round credit assignment.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Daily Dose of Data Science.