DeepSeek Trained R1 Without a Critic Network.Here Is the Actual Gradient.

2026-07-12 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

Group Relative Policy Optimisation (GRPO) is the algorithm behind DeepSeek-R1, a prominent reasoning model. GRPO replaced the widely used PPO algorithm by eliminating the expensive critic network, which previously doubled memory and compute requirements. In PPO, the critic network, having as many parameters as the policy, estimates future rewards at every token, necessitating its own forward and backward passes. By removing this component, GRPO significantly reduces computational overhead. This innovation enabled the training of models like DeepSeek-R1, which garnered significant attention for open-source reasoning capabilities. The article aims to demystify GRPO by detailing its actual gradient step-by-step, both manually and through code, moving beyond mere equation presentation.

Key takeaway

For Machine Learning Engineers optimizing large language models, understanding Group Relative Policy Optimisation (GRPO) is crucial. If you are currently using PPO, GRPO presents a compelling alternative by eliminating the resource-intensive critic network, effectively halving your compute and memory requirements. This efficiency gain, demonstrated by models like DeepSeek-R1, allows for more cost-effective and faster training of advanced reasoning capabilities. You should explore GRPO to enhance your model training workflows.

Key insights

GRPO replaces PPO by eliminating the critic network, significantly reducing computational demands for policy optimization.

Principles

• GRPO removes the critic network from PPO.
• Eliminating the critic halves compute and memory.

Method

GRPO asks "how good is this..." instead of using a critic to estimate future reward at every token.

In practice

• Train reasoning models like DeepSeek-R1.
• Reduce compute for policy optimization tasks.

Topics

• Group Relative Policy Optimisation
• PPO
• DeepSeek-R1
• Reinforcement Learning
• Critic Networks
• Computational Efficiency

Best for: AI Scientist, Machine Learning Engineer

Related on AIssential

• GRPO from Scratch: Group Relative Policy Optimization in Python — GRPO simplifies RL training by eliminating the value model, using group responses as a baseline.
• The ONLY DeepSeek GRPO/PPO video you'll EVER need (with examples and exercises) | RL Foundations — GRPO optimizes language models by adjusting rewards relative to group performance and constraining policy updates for stability.
• How DeepSeek Taught AI to Think for Itself: The Breakthrough Behind the R1 Revolution — DeepSeek's GRPO enables AI to self-correct and learn complex reasoning by comparing outputs within a group, democratizing advanced AI.
• GRPO without the maths: how an RL trainer nudges the weights — GRPO updates agent weights by comparing individual attempt scores to a group baseline, rewarding positive "advantage."
• A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism — GRPO fails to improve small web agents on mastered tasks, only helping when significant performance headroom exists.
• Reasoning RL in 2026: GRPO, DPO, RLVR, Agentic PO & Beyond — Modern reasoning RL prioritizes cheaper, critic-free, and preference-based methods over traditional PPO-style pipelines.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.