GD$^2$PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization

2026-06-15 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

GD$^2$PO, or Group-Dynamic reward-Decoupled Policy Optimization, is a new algorithm designed to mitigate multi-reward conflicts in post-training reinforcement learning for Large Language Models. Building on Group reward-Decoupled Policy Optimization (GDPO) and inspired by Dynamic sAmpling Policy Optimization (DAPO), GD$^2$PO addresses the issue where opposing reward signals cancel each other out during aggregation, hindering training efficiency. It employs a conflict-aware filtering mechanism to mask out rollouts with severe reward-wise disagreement, preserving effective RL advantages. Additionally, GD$^2$PO introduces query-level reweighting to dynamically adjust update intensity based on overall reward consensus. Experiments across multi-reward scenarios, including tool calling and human preference alignment, demonstrate that GD$^2$PO consistently and significantly outperforms existing baselines.

Key takeaway

For AI Scientists and Machine Learning Engineers developing LLMs with multi-dimensional reward functions, GD$^2$PO offers a critical advancement. Its conflict-aware filtering and query-level reweighting directly address the challenge of conflicting reward signals, which often impede training efficiency. You should consider integrating GD$^2$PO to achieve more stable and accelerated learning, particularly in complex tasks like tool calling or human preference alignment, where diverse objectives are common.

Key insights

GD$^2$PO filters conflicting reward signals to enhance RL training efficiency for LLMs.

Principles

• Decompose overall scores into independent reward groups.
• Filter out rollouts with severe reward-wise disagreement.
• Dynamically adjust update intensity based on reward consensus.

Method

GD$^2$PO uses conflict-aware filtering to mask conflicting rollouts and query-level reweighting to adjust update intensity based on overall reward consensus, accelerating RL learning.

In practice

• Apply to LLM tool calling scenarios.
• Use for human preference alignment tasks.

Topics

• Reinforcement Learning
• Large Language Models
• Multi-reward Optimization
• GD$^2$PO
• Policy Optimization
• Tool Calling

Code references

• Qwen-Applications/GD2PO

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following — MDP-GRPO stabilizes multi-constraint instruction following in LLMs by mitigating GRPO's reward-related pathologies.
• Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework — Combining two complementary internal rewards with targeted regularization prevents collapse in unsupervised RLIF.
• SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models — Dynamically reweighting MORL objectives based on real-time informativeness addresses asynchronous reward learning in LLMs.
• Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces — OPT* tasks enable scalable training and evaluation of LLM optimization reasoning by expanding search spaces without new labels.
• CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflicts — CARE-RL combines protocol-aware reward generation and capability-aware optimization to resolve multi-domain RL conflicts.
• Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning — CoUR uses LLMs, uncertainty quantification, and decoupled Bayesian optimization to create more efficient and robust RL reward functions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.