Rethinking Groups in Critic-Free RLVR

2026-06-15 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Rethinking Groups in Critic-Free RLVR introduces a novel approach to address limitations in existing critic-free Reinforcement Learning methods for post-training large language models. Current techniques typically rely on generating groups of rollouts to estimate value baselines, leading to data inefficiency, group synchronization barriers, and inflexibility with structured rollouts. This work redefines the group's core function, arguing it primarily prevents false penalties on negative samples rather than solely estimating baselines. Building on this, the authors propose "negative token filtering," a simple and effective strategy that enables stable single-rollout training. When applied to two batch-level advantage methods, this technique achieves comparable performance on reasoning tasks and stronger performance on agentic tasks compared to traditional group-based RL methods.

Key takeaway

For Machine Learning Engineers optimizing large language models with critic-free Reinforcement Learning, consider adopting negative token filtering. This method enables stable single-rollout training, directly addressing data inefficiency and synchronization barriers inherent in traditional group-based approaches. You can achieve comparable reasoning task performance and stronger results on agentic tasks, streamlining your post-training workflows and potentially reducing computational overhead.

Key insights

The core function of groups in critic-free RL is to prevent false penalties, enabling stable single-rollout training via negative token filtering.

Principles

• Group-based RL's true role is false penalty prevention.
• Single-rollout training can be stable with proper filtering.
• Data efficiency improves by avoiding group synchronization.

Method

Negative token filtering is proposed to enable stable single-rollout training in critic-free RL. This strategy prevents false penalties on negative samples, replacing the need for group-based value baseline estimation.

In practice

• Apply negative token filtering for RL post-training.
• Use single-rollout training for LLMs.
• Improve data efficiency in agentic tasks.

Topics

• Reinforcement Learning
• Large Language Models
• Critic-Free RL
• Negative Token Filtering
• Agentic Tasks
• Reasoning Tasks

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models — Dynamic sparsity schedules based on per-token actor-policy mismatch stabilize long-context RL, enabling significant speedups.
• Critic-Free, Not Bias-Free: Correcting Advantage Bias in RL from Verifier Feedback — Group-based RLHF methods for LLMs exhibit a bias that hinders learning on difficult prompts.
• GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification — GFT unifies LLM post-training by addressing SFT's limitations through group advantages and dynamic weight rectification.
• Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning — TAO-RL stabilizes agentic reinforcement learning by filtering tool-use trajectories and guiding exploration with entropy for efficient LLM policy optimization.
• RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning — POPO improves LLM reasoning by efficiently leveraging effective off-policy data in RLVR, reducing computational overhead.
• Semi-Offline Reinforcement Learning for Optimized Text Generation — The semi-offline RL paradigm optimizes text generation by blending online exploration and offline efficiency using masked observations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.