Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals

2026-05-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Reinforcement Learning with Verifiable Rewards (RLVR) often faces training instability and suboptimal convergence when scaling LLM reasoning. A systematic analysis of clipping-based GRPO-style objectives reveals that rigid clipping decisions, which discard informative signals in the near-boundary region, are a key bottleneck. To address this, Near-boundary Stochastic Rescue (NSR) is proposed as a minimal, plug-and-play modification. NSR stochastically retains slightly out-of-bound tokens, recovering lost signals that standard hard-clipping rules discard. This method, while inducing an implicit gradient decay in expectation, proves more effective than deterministic gradient decay due to its stochastic, boundary-local rescue mechanism. Extensive experiments across model sizes from 7B to 30B, including both dense and MoE architectures, validate NSR's ability to substantially improve training stability and deliver consistent performance gains over strong baselines like DAPO and GSPO.

Key takeaway

For AI Scientists and Machine Learning Engineers optimizing LLM reasoning with Reinforcement Learning with Verifiable Rewards (RLVR), you should consider integrating Near-boundary Stochastic Rescue (NSR). This plug-and-play modification directly addresses training instability caused by rigid clipping, which often discards valuable near-boundary signals. By stochastically recovering these signals, NSR consistently improves stability and performance over baselines like DAPO and GSPO, offering a straightforward path to more robust RLVR training across diverse model architectures.

Key insights

Rigid clipping in RLVR discards valuable near-boundary signals, causing instability; stochastic recovery improves performance.

Principles

• Hard clipping can discard informative near-boundary signals.
• Stochastic signal recovery outperforms deterministic gradient decay.

Method

NSR stochastically retains slightly out-of-bound tokens in clipping-based GRPO-style objectives to recover lost signals.

In practice

• Apply NSR to stabilize RLVR training for LLM reasoning.
• Integrate NSR as a plug-and-play modification in existing RLVR setups.

Topics

• Reinforcement Learning
• LLM Reasoning
• Verifiable Rewards
• Clipping Algorithms
• Training Stability
• Stochastic Optimization

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently — RLVR's outcome-based rewards enable LLMs to learn efficient backtracking, unlike SFT, leading to exponential inference compute gains.
• CL-CLIP: CLIP-Based Continual Learning Framework with Cost-Volume Category Decoupling for Object Detection — CLIP-based continual object detection benefits from category decoupling via a frozen image-text similarity cost volume.
• Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers — Gradient clipping theoretically justifies ASGD robustness by removing maximum delay dependence, even with heavy-tailed noise.
• Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging — RLVR's sparse, dispersed parameter updates create a "sparsity curse" that prevents effective model merging, unlike SFT models.
• MARS: Margin-Aware Reward-Modeling with Self-Refinement — MARS improves reward modeling by adaptively augmenting ambiguous preference data to enhance model robustness.
• Understanding Diversity Collapse in RLVR via the Lens of Overtraining — Diversity collapse in RLVR is overtraining, concentrating probability mass on already solved problems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.