The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

A recent study reveals that applying dense per-step prediction rewards to Large Language Model (LLM) agents trained with Group-Normalized Reinforcement Learning (GRPO) on ALFWorld environments leads to a "dark room" pathology. Specifically, across Qwen3-1.7B, Qwen3-4B, and Qwen3-8B models, a potential-based prediction reward under GRPO causes agents to achieve 100% prediction accuracy but 0% task success, with episode lengths fixed at the horizon. The core issue is identified as GRPO's standard deviation normalization; its removal transforms the reward from catastrophic (0% success) to baseline performance. The research posits that GRPO's z-scoring amplifies within-group variance in all-fail groups, making signals with variance that decays by mastery "amplifier-safe." Furthermore, the auxiliary-loss channel demonstrated a ~20-point gain in performance, even with shuffled-gold placebo labels, outperforming the reward channel which was at best neutral.

Key takeaway

For Machine Learning Engineers developing LLM agents, especially with GRPO, you should critically re-evaluate dense prediction reward strategies. Your current approach might lead to policy collapse, as GRPO's standard deviation normalization can create degenerate states. Instead, consider disabling GRPO's std normalization or exploring auxiliary-loss channels, which demonstrated significant performance gains even with imperfect labels. This shift can prevent "dark room" pathologies and improve task success.

Key insights

GRPO's z-scoring amplifies dense reward variance, causing LLM agent collapse; auxiliary losses are more effective.

Principles

• GRPO's std normalization can destroy policies with dense prediction rewards.
• Signals with variance decaying by mastery are "amplifier-safe."
• Auxiliary-loss channels can outperform reward channels.

In practice

• Avoid GRPO's std normalization with dense prediction rewards.
• Prioritize auxiliary-loss channels for LLM agent training.
• Evaluate reward signals for variance decay by mastery.

Topics

• LLM Agents
• Reinforcement Learning
• GRPO
• Reward Shaping
• Auxiliary Losses
• ALFWorld

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization — CW-GRPO improves LLM search agents by using an LLM judge for fine-grained, per-round credit assignment.
• Reward hacking in Reinforcement learning — Reward hacking is an inevitable outcome of imperfect objective functions in RL, especially with powerful LLM optimizers.
• Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation — AdaGRPO selectively applies RL rewards based on policy uncertainty and ranker discriminability to improve generative recommendation.
• GRPO in Production: The Failure Modes Nobody Writes About — GRPO, a popular LLM post-training algorithm, has three silent failure modes—advantage collapse, entropy collapse, and KL drift—that stall training.
• Overcoming reward signal challenges: Verifiable rewards-based reinforcement learning with GRPO on SageMaker AI — RLVR with GRPO and few-shot examples significantly improves LLM accuracy on verifiable tasks.
• Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair — Signal reshaping improves GRPO performance in code-agent RL by enhancing feedback quality and comparability.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.