The ONLY DeepSeek GRPO/PPO video you'll EVER need (with examples and exercises) | RL Foundations

2025-02-14 · Source: Depth First · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, extended

Summary

The DeepSeek R1 model utilizes a Group Relative Policy Optimization (GRPO) training objective, which is a reinforcement learning (RL) method designed to optimize language models. This objective function, detailed in equation one of the R1 paper, incorporates several key components, including an expectation term for averaging scores across samples, a policy term representing the language model's probability of generating a response, and a crucial "advantage" term. The advantage is computed by adjusting the raw reward (0 for incorrect, 1 for correct) based on the average performance and standard deviation of a group of responses to the same question, making it "group relative." The objective also includes a clipping mechanism and a minimum function to constrain policy updates, preventing excessive divergence from previous states. Additionally, a KL Divergence term ensures the model remains similar to a frozen reference policy, DeepSeek V3, balancing performance improvement with policy stability. This complex objective addresses the inherent noisiness and sparse feedback challenges of reinforcement learning compared to supervised learning.

Key takeaway

For Machine Learning Engineers developing or fine-tuning large language models with reinforcement learning, understanding the GRPO objective is critical. You should prioritize implementing mechanisms like group-relative advantage and policy clipping to manage the inherent noise and sparse feedback of RL, ensuring stable and effective model optimization. This approach helps prevent reward hacking and maintains a balance between performance gains and policy consistency, crucial for robust LLM development.

Key insights

GRPO optimizes language models by adjusting rewards relative to group performance and constraining policy updates for stability.

Principles

• Reinforcement learning is inherently noisier than supervised learning.
• Conservatism in policy updates is crucial in RL.
• Group-relative advantage reduces RL training costs.

Method

GRPO computes advantage by subtracting group average reward and dividing by standard deviation. It then clips probability ratios and uses a min function to constrain updates, balancing performance with policy similarity via KL divergence.

In practice

• Use group-relative advantage for cost-effective RL.
• Implement clipping and min functions to stabilize policy updates.
• Incorporate KL divergence to maintain similarity to a base model.

Topics

• Group Relative Policy Optimization
• Reinforcement Learning
• Advantage Function
• Policy Regularization
• DeepSeek R1

Best for: Machine Learning Engineer, Deep Learning Engineer, AI Student

Related on AIssential

• Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment — P-GRPO enhances LLM alignment by normalizing advantages against preference-group-specific reward histories.
• Rethinking the Divergence Regularization in LLM RL — DRPO improves LLM RL stability by replacing hard trust-region masks with a smooth, corrective divergence regularizer.
• Verifiable Rewards and GRPO — RLVR and GRPO offer a cost-effective, robust alternative to RLHF for tasks with verifiable correctness by eliminating learned reward models and critics.
• AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO — Adaptive GRPO enhances text-to-image flow models by dynamically matching training prompts to model capability and fusing local-global advantage estimates.
• Reasoning RL in 2026: GRPO, DPO, RLVR, Agentic PO & Beyond — Modern reasoning RL prioritizes cheaper, critic-free, and preference-based methods over traditional PPO-style pipelines.
• FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning — Feedback-driven bi-objective reinforcement learning significantly boosts LLM training efficiency and performance by synergizing exploitation and exploration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Depth First.