Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games

2026-02-16 · Source: cs.MA updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Bench-MFG is a new benchmark suite designed to standardize the evaluation of learning algorithms in stationary Mean Field Games (MFGs), addressing the current fragmentation in research. The suite introduces a taxonomy of problem classes, including No-Interaction, Contractive, Lasry-Lions Monotone, Potential, and Dynamics-Coupled games, providing prototypical environments for each. It also features MF-Garnets, a method for procedurally generating random MFG instances to enable robust statistical testing. Researchers benchmarked various learning algorithms, such as Fixed Point Iteration, Policy Iteration, Fictitious Play, Online Mirror Descent (OMD), and a novel black-box exploitability minimizer called MF-PSO, across these diverse environments. The implementation leverages JAX for a 2000x speedup compared to classical Python, with code available on GitHub. Empirical results inform guidelines for future experimental comparisons in the field.

Key takeaway

For research scientists developing or evaluating Mean Field Game algorithms, you should adopt the Bench-MFG suite to ensure robust and reproducible comparisons. This means testing your algorithms across the provided taxonomy of MFG classes and utilizing MF-Garnets for statistical validation. Prioritize computationally efficient implementations, such as those in JAX, to facilitate exhaustive hyperparameter tuning and comprehensive regime testing, thereby strengthening the empirical rigor of your findings.

Key insights

Bench-MFG standardizes Mean Field Game evaluation through a diverse benchmark suite and procedural generation.

Principles

• MFG evaluation requires diverse problem classes.
• Procedural generation enables rigorous statistical testing.
• Exploitability minimization is a valid objective.

Method

Bench-MFG defines a taxonomy of MFG problem classes, provides prototypical environments, and introduces MF-Garnets for generating random instances. It benchmarks algorithms like MF-PSO by minimizing exploitability.

In practice

• Use JAX for significant computational speedups.
• Compare against Fixed Point, Fictitious Play, OMD, and MF-PSO.
• Test algorithms across diverse MFG regimes and parameters.

Topics

• Mean Field Games
• Reinforcement Learning
• Benchmark Suite
• Multi-Agent Systems
• MF-PSO

Code references

• lorenzomagnino/Bench-MFG
• jax-ml/jax

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• In game theory, generalists sometimes win out over specialists — General-purpose policy gradient methods can surprisingly outperform specialized game-theoretic algorithms in imperfect-information games.
• Kaggle Conversations with Alex Shaw: Designing a Robust Eval Framework for the $1M Konwinski Prize — Robust AI agent evaluation requires dynamic, contamination-free benchmarks and standardized, flexible tooling.
• Mean Flow Policy Optimization — MeanFlow models offer efficient policy representation for RL, outperforming diffusion models in speed and comparable in performance.
• Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences — Large-scale, out-of-distribution benchmarks reveal state-tracking failures in models that scale alone conceals.
• Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining — Complex multi-agent benchmarks reveal LLM strategic reasoning limitations in integrated economic games.
• Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Play — Multi-Agent Fictitious Play (MAFP) resolves interdependent decision-making by modeling stakeholder stances as agents in an equilibrium-seeking process.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.MA updates on arXiv.org.