Training Agents 3: Reinforcement Learning

2026-07-22 · Source: HuggingFace · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, quick

Summary

Training Agents, Session 3 introduces Group Relative Policy Optimization (GRPO) as a method for training agents on outcomes, building upon models from earlier supervised fine-tuning (SFT) sessions. GRPO operates by sampling multiple completions per prompt, scoring them with a reward function, and then utilizing the group's relative scores as the training signal. This approach uniquely bypasses the need for a value model, reward model, or critic. The session will detail GRPO's operational mechanics, including how relative advantages replace a critic, guide participants in writing verifiable reward functions for agent tasks, and demonstrate running TRL GRPO on Hugging Face Jobs. It will also cover interpreting training curves (reward, KL, completion length) and identifying common failure modes such as reward hacking, collapse, and length gaming.

Key takeaway

For AI Engineers developing agents that require outcome-based training beyond imitation, exploring Group Relative Policy Optimization (GRPO) is critical. This method allows you to train models on actual results using relative scoring, bypassing the complexity of training separate critics or reward models. You should focus on designing robust, verifiable reward functions and be prepared to diagnose common failure modes like reward hacking to ensure effective agent performance.

Key insights

GRPO enables outcome-based agent training using relative scores from multiple completions, eliminating the need for a separate critic or reward model.

Principles

• GRPO trains on outcomes, not just imitation.
• Relative scores from groups replace explicit critics.
• Verifiable reward functions are crucial for agents.

Method

GRPO involves sampling multiple completions per prompt, scoring them with a reward function, and using the group's relative scores as the training signal within TRL.

In practice

• Run TRL GRPO on Hugging Face Jobs.
• Analyze reward, KL, and length training curves.
• Identify reward hacking and length gaming.

Topics

• Reinforcement Learning
• Group Relative Policy Optimization
• Agent Training
• Reward Functions
• Hugging Face TRL
• Policy Optimization

Code references

• burtenshaw/training-agents

Best for: Machine Learning Engineer, AI Engineer, AI Scientist

Related on AIssential

• GRPO without the maths: how an RL trainer nudges the weights — GRPO updates agent weights by comparing individual attempt scores to a group baseline, rewarding positive "advantage."
• Overcoming reward signal challenges: Verifiable rewards-based reinforcement learning with GRPO on SageMaker AI — RLVR with GRPO and few-shot examples significantly improves LLM accuracy on verifiable tasks.
• A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism — GRPO fails to improve small web agents on mastered tasks, only helping when significant performance headroom exists.
• MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following — MDP-GRPO stabilizes multi-constraint instruction following in LLMs by mitigating GRPO's reward-related pathologies.
• The ONLY DeepSeek GRPO/PPO video you'll EVER need (with examples and exercises) | RL Foundations — GRPO optimizes language models by adjusting rewards relative to group performance and constraining policy updates for stability.
• Agentic RL: Environments, Trajectories, and the Training — Training multi-step LLM agents requires scoring full interaction trajectories using an LLM-as-a-judge within a GRPO framework.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by HuggingFace.