Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning

2026-05-11 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

This paper introduces Decentralized Diffusion Policy Learning (DDPL), a novel approach to cooperative multi-agent reinforcement learning (MARL) that enhances exploration by parameterizing agent policies with denoising diffusion probabilistic models (DDPMs). The authors demonstrate that traditional Gaussian policies, commonly used in decentralized softmax policy gradient (DecSPG) algorithms, severely limit exploration and lead to suboptimal equilibria, a problem that worsens with an increasing number of agents. DDPL addresses this by enabling efficient online training of diffusion policies through importance sampling score matching (ISSM), a new method with theoretical guarantees. Empirical evaluations on continuous-action MARL benchmarks, including multi-agent particle environment (MPE), multi-agent MuJoCo (MaMuJoCo), IsaacLab, and JAX-reimplemented StarCraft Multi-Agent Challenge (SMAX), show consistent performance improvements and better sample efficiency compared to existing baselines like HAPPO, MAPPO, and HASAC.

Key takeaway

For research scientists developing cooperative MARL systems, DDPL offers a robust solution to the exploration challenge. You should consider adopting diffusion models for policy parameterization, as they demonstrably capture multi-modal action distributions crucial for discovering optimal coordination patterns. This approach, particularly with the efficient ISSM training method, can prevent premature convergence to suboptimal equilibria, especially in environments with a high number of agents, leading to significantly improved performance and sample efficiency.

Key insights

Diffusion policies enable multi-modal exploration in MARL, outperforming restrictive Gaussian policies.

Principles

• Policy expressiveness is critical for effective MARL exploration.
• Gaussian policies hinder exploration in multi-agent settings.
• Multi-modal action distributions improve discovery of high-reward equilibria.

Method

DDPL parameterizes decentralized policies with DDPMs and trains them online using Importance Sampling Score Matching (ISSM), which moves diffusion policies toward energy-based target distributions without requiring target samples.

In practice

• Apply DDPL to continuous-action MARL tasks for enhanced exploration.
• Use ISSM for efficient online training of diffusion policies.
• Consider diffusion models for complex, multi-modal action distributions.

Topics

• Decentralized Diffusion Policy Learning
• Cooperative Multi-agent Reinforcement Learning
• Denoising Diffusion Probabilistic Models
• Importance Sampling Score Matching
• Multi-modal Policy Exploration

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Individual Control Barrier Functions-Guided Diffusion Model for Safe Offline Multi-Agent Reinforcement Learning — Integrating neural individual control barrier functions into diffusion models enables safe offline multi-agent reinforcement learning.
• Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning — Dynamically adjusting exploration intensity globally and per-agent based on reward signal quality improves MARL performance.
• Discrete Diffusion for Complex and Congested Multi-Agent Path Finding with Sparse Social Attention — DiffLNS combines discrete diffusion for initial plan generation with LNS2 for robust multi-agent pathfinding repair.
• Low-Complexity Policy Tessellations in Structured Markov Decision Processes — Optimal policies in structured MDPs induce simpler decision boundaries than their high-dimensional value functions.
• Multi-Agent Model-Based Reinforcement Learning with Joint State-Action Learned Embeddings — MMSA combines model-based MARL with joint state-action representation learning for enhanced coordination and sample efficiency.
• Reinforcement Learning from Denoising Feedback — RLDF enhances dLLM reasoning by efficiently estimating policy loss via weighted sampling of denoising feedback and clean state prediction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.