A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization

2026-06-15 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new analysis investigates the instability issues in GRPO-style optimization within Reinforcement Learning with Verifiable Rewards (RLVR), a method used to enhance language model reasoning. This research employs token-level gradient dynamics to develop a taxonomy, revealing that optimization stability is jointly determined by the advantage sign and the token distribution under the current policy. Based on this finding, the authors introduce Winner Advantage Policy Optimization (WAPO), a straightforward online clipped policy-gradient objective. WAPO specifically updates only on completions that exhibit a positive advantage. Evaluated across mathematical reasoning and multi-hop QA benchmarks, WAPO consistently improves training stability and either matches or surpasses the performance of existing baselines across various model families. Full code is available on GitHub.

Key takeaway

For Machine Learning Engineers developing language models with RLVR, consider integrating Winner Advantage Policy Optimization (WAPO) to mitigate training instability. Your GRPO-style optimization efforts can benefit from WAPO's approach of updating only on positive-advantage completions, which has demonstrated improved stability and competitive performance on mathematical reasoning and multi-hop QA benchmarks. Explore the provided GitHub repository to implement WAPO in your current projects.

Key insights

RLVR optimization stability hinges on advantage sign and token distribution, addressed by WAPO's positive-advantage updates.

Principles

• RLVR stability depends on advantage sign.
• Token distribution impacts policy updates.
• Positive-advantage updates enhance stability.

Method

Winner Advantage Policy Optimization (WAPO) is an online clipped policy-gradient objective that updates only on completions with a positive advantage.

In practice

• Apply WAPO for stable RLVR training.
• Use WAPO on mathematical reasoning tasks.
• Implement WAPO for multi-hop QA models.

Topics

• Reinforcement Learning with Verifiable Rewards
• Policy Optimization
• Gradient Dynamics
• Language Models
• Mathematical Reasoning
• Multi-hop QA

Code references

• layer6ai-labs/wapo

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Overcoming reward signal challenges: Verifiable rewards-based reinforcement learning with GRPO on SageMaker AI — RLVR with GRPO and few-shot examples significantly improves LLM accuracy on verifiable tasks.
• Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity — RLVR's diversity collapse stems from objective indifference and on-policy sampling, which UCPO addresses by promoting uniform probability across correct solutions.
• AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO — Adaptive GRPO enhances text-to-image flow models by dynamically matching training prompts to model capability and fusing local-global advantage estimates.
• Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization — GNDPO stabilizes on-policy distillation for MLLM reasoning by normalizing KL scores into batch-level relative advantages.
• From GRPO to SAMPO: Solving Training Collapse in Agentic RL — SAMPO stabilizes agentic RL training by optimizing four key policy dimensions to prevent catastrophic gradient issues.
• The ONLY DeepSeek GRPO/PPO video you'll EVER need (with examples and exercises) | RL Foundations — GRPO optimizes language models by adjusting rewards relative to group performance and constraining policy updates for stability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.