GRPO in Production: The Failure Modes Nobody Writes About

2026-06-22 · Source: Deep Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, medium

Summary

GRPO (Group Relative Policy Optimization) is now the default RL algorithm for post-training Large Language Models, adopted by models like DeepSeek-R1, Nemotron 3 Super, and Qwen-Math derivatives. It eliminates the PPO critic network, cutting training compute by nearly half. However, GRPO's group-relative advantage computation has three silent failure modes: advantage collapse, entropy collapse, and KL drift. Advantage collapse, the most common, occurs when all responses in a group receive uniform rewards, leading to zero gradient signal. Entropy collapse reduces response diversity, while KL drift results from an improperly tuned KL penalty. The article also identifies a fourth issue: sample-level loss normalization, which biases against longer chain-of-thought responses. DAPO (Dynamic Sampling Policy Optimization) offers algorithmic fixes for these issues, including dynamic sampling, an asymmetric KL clip, decoupled KL, and token-level normalization.

Key takeaway

For MLOps Engineers or AI Scientists deploying GRPO for LLM post-training, you must actively monitor for advantage collapse, entropy collapse, and KL drift. Implement DAPO's dynamic sampling, asymmetric KL clipping, decoupled KL, and token-level normalization to prevent training stalls and ensure robust model improvement. Your reward model's accuracy and the initial SFT checkpoint's entropy are critical upstream factors impacting GRPO's success.

Key insights

GRPO, a popular LLM post-training algorithm, has three silent failure modes—advantage collapse, entropy collapse, and KL drift—that stall training.

Principles

• GRPO's group-relative advantage is prone to collapse with uniform rewards.
• Policy entropy is critical for maintaining generation diversity.
• KL penalty tuning impacts policy drift and training signal.

Method

DAPO addresses GRPO failures via dynamic sampling to filter uniform reward groups, an asymmetric KL clip to preserve entropy, decoupled KL from reward, and token-level loss normalization to reinforce longer responses.

In practice

• Monitor entropy and advantage variance as key training metrics.
• Audit your reward model before diagnosing GRPO training issues.
• Check SFT model entropy before starting RL fine-tuning.

Topics

• GRPO
• LLM Post-training
• Policy Optimization
• Advantage Collapse
• Entropy Collapse
• DAPO Algorithm

Best for: Machine Learning Engineer, AI Scientist, MLOps Engineer

Related on AIssential

• GRPO++: Tricks for Making RL Actually Work — Optimizing GRPO for LLM reasoning requires addressing inherent biases and system-level mismatches to achieve stable and efficient training.
• MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following — MDP-GRPO stabilizes multi-constraint instruction following in LLMs by mitigating GRPO's reward-related pathologies.
• STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability — STARE rebalances token-level credit in GRPO via surprisal-guided reweighting and a closed-loop gate to prevent policy entropy collapse.
• ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information — Asynchronous LLM post-training can be stabilized by adaptively scaling current-policy loss, removing behavior-policy dependencies.
• AOI: Turning Failed Trajectories into Training Signals for Autonomous Cloud Diagnosis — Failed diagnostic trajectories are valuable training signals for improving autonomous SRE agents.
• Post-Training Matters More Than Pretraining Now: SFT, RLHF, DPO, and GRPO. — Post-training methods are now critical for advancing LLM capabilities beyond mere imitation to complex reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Deep Learning on Medium.