GRPO Does Not Close the Multi-Agent Coordination Gap

2026-06-05 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Multiagent Systems · Depth: Expert, quick

Summary

A study evaluating large language models' multi-agent coordination capabilities using the dining philosophers problem found that current systems struggle, and Group Relative Policy Optimization (GRPO) does not close this performance gap. Across 630 episodes with seven models, frontier closed-source systems achieved mean rewards of 0.45 to 0.87, while Mistral-Small 24B reached 0.83 to 0.99, and Qwen3-14B scored 0.13 to 0.35. A Welch's t-test on GRPO's impact at five philosophers yielded p = 0.66 and Hedges' g of -0.11, indicating no statistically significant improvement. Training rewards for 8B and 14B models peaked at step nine before declining, making the default step 15 checkpoint suboptimal. The four-term reward function also allowed a degenerate maximum at zero actions, exploited by DeepSeek-R1-Distill-Qwen-7B and Mistral-Small 24B. The primary bottleneck for open-weight 14B models is identified as training methodology, specifically reward shaping, checkpoint discipline, and curriculum design, rather than computational power.

Key takeaway

For AI Scientists and Machine Learning Engineers developing LLMs for multi-agent coordination, you should recognize that current models and GRPO do not adequately solve the coordination gap. Your focus must shift from raw compute to refining training methodology. Specifically, prioritize designing robust reward functions that avoid degenerate maxima, implementing disciplined checkpoint selection beyond the final step, and developing effective curricula across problem scales to improve multi-agent performance.

Key insights

Large language models struggle with multi-agent coordination, and GRPO does not improve it; training methodology is the core bottleneck.

Principles

• Reward functions can admit degenerate maxima.
• Optimal model performance requires careful checkpoint selection.
• Effective training methodology is crucial for multi-agent LLMs.

In practice

• Design reward functions to prevent zero-action maxima.
• Implement checkpoint discipline beyond final step saves.
• Apply curriculum learning for multi-agent problem scales.

Topics

• Multi-agent Systems
• Large Language Models
• Reinforcement Learning
• Reward Shaping
• Checkpoint Management
• Curriculum Learning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization — CW-GRPO improves LLM search agents by using an LLM judge for fine-grained, per-round credit assignment.
• Dynamic Coordination Strategy Selection for Enterprise Multi-Agent Systems — Dynamic strategy selection for enterprise multi-agent systems is a near-best routing heuristic, not a deterministic winner-selection law.
• From GRPO to SAMPO: Solving Training Collapse in Agentic RL — SAMPO stabilizes agentic RL training by optimizing four key policy dimensions to prevent catastrophic gradient issues.
• MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following — MDP-GRPO stabilizes multi-constraint instruction following in LLMs by mitigating GRPO's reward-related pathologies.
• Scaling up multi-agent systems: an interview with Minghong Geng — Hierarchical and hybrid designs are key to scaling multi-agent systems for complex, long-horizon problems.
• Post-Training Matters More Than Pretraining Now: SFT, RLHF, DPO, and GRPO. — Post-training methods are now critical for advancing LLM capabilities beyond mere imitation to complex reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.