Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework

2026-05-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new multi-reward Reinforcement Learning from Internal Feedback (RLIF) framework addresses limitations of existing single-reward RLIF methods, which often suffer from reward hacking, entropy collapse, and degraded reasoning structures. This novel approach decomposes the training signal into two complementary components: an answer-level reward derived from cluster voting and a completion-level reward based on token-wise self-certainty. To robustly combine these signals, the framework applies GDPO-based normalization, mitigating reward-scale imbalance. Furthermore, it introduces KL-Cov regularization, specifically targeting low-entropy token distributions to preserve exploration and prevent late-stage collapse. Evaluated across mathematical reasoning and code-generation benchmarks, the method demonstrates improved stability and robustness compared to prior unsupervised RL techniques, achieving performance levels close to supervised Reinforcement Learning with Verifiable Rewards (RLVR) methods.

Key takeaway

For Machine Learning Engineers developing large language models for complex reasoning tasks without external supervision, this multi-reward RLIF framework offers a robust solution. You should consider integrating complementary internal rewards, such as answer-level cluster voting and token-wise self-certainty, into your training pipelines. Implementing GDPO-based normalization and KL-Cov regularization can significantly enhance stability, prevent entropy collapse, and improve long-horizon reasoning performance, bringing unsupervised methods closer to supervised benchmarks.

Key insights

Combining two complementary internal rewards with targeted regularization prevents collapse in unsupervised RLIF.

Principles

• Decompose training signals into complementary components.
• Normalize reward scales to reduce imbalance.
• Regularize low-entropy distributions to preserve exploration.

Method

Propose a multi-reward RLIF framework using answer-level (cluster voting) and completion-level (token-wise self-certainty) rewards, normalized via GDPO, and regularized with KL-Cov to prevent entropy collapse.

In practice

• Apply multi-reward RLIF for stable long-horizon reasoning.
• Use GDPO-based normalization for diverse reward signals.
• Implement KL-Cov regularization to prevent entropy collapse.

Topics

• Reinforcement Learning from Internal Feedback
• Large Language Models
• Multi-reward Systems
• Entropy Regularization
• Mathematical Reasoning
• Code Generation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Efficient Exploration, Reasoning, and Training-Free MTP — Efficiency gains in LLMs are achievable through online RLHF, balanced reasoning control, and embedding-space multi-token prediction.
• Maximum Entropy Semi-Supervised Inverse Reinforcement Learning — MESSI improves Inverse Reinforcement Learning by using unsupervised data to refine reward functions via a pairwise similarity penalty.
• Beyond Mode Collapse: Distribution Matching for Diverse Reasoning — Distribution matching prevents mode collapse in on-policy RL by promoting diverse exploration, improving reasoning capabilities.
• Uncertainty-Aware Reward Modeling for Stable RLHF — Equipping RLHF reward models with uncertainty awareness prevents reward hacking and improves alignment.
• RREDCoT: Segment-Level Reward Redistribution for Reasoning Models — RREDCoT uses the language model itself for efficient, segment-level reward redistribution in Chain-of-Thought RL fine-tuning.
• GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization — GD$^2$PO filters conflicting reward signals to enhance RL training efficiency for LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.