GRPO without the maths: how an RL trainer nudges the weights

2026-06-17 · Source: Alex Strick van Linschoten · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Novice, long

Summary

Group Relative Policy Optimisation (GRPO) is an algorithm commonly used by the trainer component to nudge the model (policy) in the right direction during reinforcement learning. It operates by attempting a problem multiple times, calculating a "baseline" (average score of all attempts), and then determining an "advantage" for each attempt (distance from the baseline). Policy weights are updated based on positive advantage, as shown in a simplified pseudocode example. While GRPO grades the whole attempt, making it less suitable for long-running tasks with complex credit assignment, Proximal Policy Optimization (PPO) uses a critic model for fine-grained, token-level advantage estimation. This PPO approach, highlighted by z.ai's GLM-5.2 model released yesterday, is better suited for long-horizon agentic tasks, as GRPO can hit performance ceilings on such problems.

Key takeaway

For AI Engineers developing agentic systems, if your tasks involve long execution traces or require fine-grained credit assignment, you should prioritize Proximal Policy Optimization (PPO) over Group Relative Policy Optimisation (GRPO). While GRPO offers a simpler, group-based update mechanism, it may hit performance ceilings on complex, multi-step problems. Evaluate your task horizon carefully; PPO's critic-based, token-level advantage estimation is crucial for scaling to intricate agent behaviors.

Key insights

GRPO updates agent weights by comparing individual attempt scores to a group baseline, rewarding positive "advantage."

Principles

• GRPO evaluates whole attempts, not specific steps.
• Task difficulty impacts learning signal in RL.
• Critic models are necessary for long-horizon RL tasks.

Method

GRPO samples a group of attempts, scores them, calculates a baseline (group average), then nudges policy weights based on each attempt's "advantage" (score minus baseline).

In practice

• Use GRPO for short, well-defined agentic tasks.
• Consider PPO for long-horizon tasks requiring fine-grained credit assignment.
• Monitor task difficulty to ensure sufficient learning signal.

Topics

• Group Relative Policy Optimisation
• Proximal Policy Optimization
• Reinforcement Learning
• Agentic AI
• Policy Optimization
• Credit Assignment

Code references

• strickvl/hinbox

Best for: AI Engineer, Machine Learning Engineer, AI Student

Related on AIssential

• The ONLY DeepSeek GRPO/PPO video you'll EVER need (with examples and exercises) | RL Foundations — GRPO optimizes language models by adjusting rewards relative to group performance and constraining policy updates for stability.
• Training Agents 3: Reinforcement Learning — GRPO enables outcome-based agent training using relative scores from multiple completions, eliminating the need for a separate critic or reward model.
• Proximal Policy Optimization — Proximal Policy Optimization (PPO) stabilizes policy gradient methods by constraining update steps within a "trust region" to prevent performance collapse.
• Generalized Per-Agent Advantage Estimation for Multi-Agent Policy Optimization — GPAE improves multi-agent reinforcement learning by providing precise per-agent advantage estimation and stable off-policy learning.
• A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism — GRPO fails to improve small web agents on mastered tasks, only helping when significant performance headroom exists.
• Reasoning RL in 2026: GRPO, DPO, RLVR, Agentic PO & Beyond — Modern reasoning RL prioritizes cheaper, critic-free, and preference-based methods over traditional PPO-style pipelines.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Alex Strick van Linschoten.