GRPO from Scratch: Group Relative Policy Optimization in Python

2026-07-20 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Group Relative Policy Optimization (GRPO) offers a memory-efficient alternative to Proximal Policy Optimization (PPO) for training language models. While PPO requires four active components (policy, reference, reward, value) with a 42 GB footprint for its three trainable weight copies, GRPO eliminates the value model, reducing the active footprint to 28 GB. GRPO achieves this by generating multiple responses to a single prompt, normalizing rewards within this group to calculate advantages, and directly applying the PPO clipping objective without needing a separate critic or value function. This method simplifies the architecture, making the group itself the baseline. DeepSeek-R1, a reasoning model, was notably trained using GRPO, demonstrating its utility for models that require complex reasoning without extensive reward model infrastructure.

Key takeaway

For Machine Learning Engineers optimizing large language model training, GRPO presents a compelling option to reduce memory overhead. By eliminating the value model, you can significantly lower the active footprint to 28 GB compared to PPO's 42 GB. Consider adopting GRPO, as demonstrated by DeepSeek-R1, to simplify your reinforcement learning architecture and potentially train complex reasoning models more efficiently without a dedicated reward model rollout infrastructure.

Key insights

GRPO simplifies RL training by eliminating the value model, using group responses as a baseline.

Principles

• The group serves as the baseline for reward normalization.
• Directly apply PPO clipping objective without a critic.

Method

Generate G responses to a prompt, normalize rewards within the group to compute advantages, then apply the PPO clipping objective directly.

In practice

• Train reasoning models without separate reward infrastructure.
• Reduce active memory footprint from 42 GB to 28 GB.

Topics

• Group Relative Policy Optimization
• Proximal Policy Optimization
• Reinforcement Learning
• Language Models
• DeepSeek-R1
• Memory Optimization

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• DeepSeek Trained R1 Without a Critic Network.Here Is the Actual Gradient. — GRPO replaces PPO by eliminating the critic network, significantly reducing computational demands for policy optimization.
• The ONLY DeepSeek GRPO/PPO video you'll EVER need (with examples and exercises) | RL Foundations — GRPO optimizes language models by adjusting rewards relative to group performance and constraining policy updates for stability.
• A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism — GRPO fails to improve small web agents on mastered tasks, only helping when significant performance headroom exists.
• GRPO without the maths: how an RL trainer nudges the weights — GRPO updates agent weights by comparing individual attempt scores to a group baseline, rewarding positive "advantage."
• Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment — P-GRPO enhances LLM alignment by normalizing advantages against preference-group-specific reward histories.
• Mixed-Policy GRPO for Text-to-SQL with Off-Policy Data Generation — Mixed-policy GRPO with off-policy data generation significantly enhances text-to-SQL reasoning by diversifying training trajectories.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.