CoffeeBench: A Long-Term Task Benchmark for LLM Agents in a Multi-Agent Economic Environment

2026-06-26 · Source: Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, medium

Summary

Sakana AI and Azusa Audit Corporation released "CoffeeBench" on June 26, 2026, a new benchmark designed to evaluate the long-term management capabilities of LLM agents within a multi-agent economic environment. This simulation tasks agents with operating a coffee roasting business for 90 days, aiming to generate profits by trading with virtual farmers and retailers. Experiments with models like GPT-5.5 and Claude Opus 4.7 revealed significant performance disparities, with high-performing agents actively engaging in negotiations and communication. Notably, Claude Haiku 4.5 exhibited a "stagnation phenomenon," analyzing situations but failing to act. CoffeeBench also aims to serve as a framework for studying agent behavior, including potential fraudulent activities, anticipating future LLM agent roles in corporate management. The research is scheduled for presentation at the ICML2026 Workshop Failure Modes in Agentic AI.

Key takeaway

For Directors of AI/ML evaluating LLM agents for corporate management, recognize that current models exhibit significant variability in long-term decision-making and proactive engagement. Your evaluation strategy should incorporate multi-agent economic simulations like CoffeeBench to uncover model-specific behavioral patterns, including potential "thought-action" discrepancies. Prioritize agents demonstrating consistent proactive communication and transaction execution to ensure robust, profit-generating operations in complex supply chains.

Key insights

The benchmark reveals LLM agents' long-term decision-making capabilities and behavioral patterns in complex economic simulations.

Principles

• Proactive communication drives profit in multi-agent economies.
• Long-term tasks expose "thought-action" discrepancies in LLMs.
• Economic simulations can reveal agent fraud mechanisms.

Method

CoffeeBench simulates a 90-day coffee supply chain with six LLM agents (farmers, roasters, retailers) interacting via tools for transactions, negotiations, and inventory management to maximize net profit.

In practice

• Evaluate LLM agents for corporate management roles.
• Study agent cooperation, competition, and deviant behaviors.
• Research auditing and governance methods for agentic AI.

Topics

• CoffeeBench
• LLM Agents
• Multi-Agent Systems
• Economic Simulation
• Supply Chain Management
• Agent Governance

Code references

• SakanaAI/CoffeeBench

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies — Active communication is crucial for LLM agents to succeed in long-horizon, multi-agent economic simulations.
• EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent — EComAgentBench reveals current LLM shopping agents struggle significantly with uncovering hidden user intent across long-horizon tasks.
• Benchmarking Open-Ended Multi-Agent Coordination in Language Agents — Open-ended multi-agent coordination is a distinct bottleneck for frontier LLMs, measurable by the $alem$ benchmark.
• GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows — GTA-2 benchmarks LLM agents on real-world atomic tool use and complex, open-ended workflows, revealing significant capability gaps.
• AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition — LLM agents universally suffer significant performance degradation under realistic noise, especially from tool failures.
• Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents — APB diagnostically evaluates LLM agent planning across diverse scenarios, revealing systematic weaknesses and guiding refinement.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Blog.