PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Data Science & Analytics · Depth: Expert, quick

Summary

PolyWorkBench is a new benchmark designed to evaluate large language model (LLM) agents on multilingual long-horizon workplace workflows. It addresses the gap where most existing benchmarks implicitly assume monolingual execution, despite real-world applications often requiring multilingual inputs and outputs. PolyWorkBench comprises 67 tasks across five domains, including commerce, knowledge work, legal analysis, localization, and manufacturing. Agents must process diverse multilingual inputs, perform iterative reasoning, invoke external tools, and generate structured outputs. The benchmark employs a hybrid evaluation framework, combining structural grading, executable verification, and LLM-based semantic assessment. Empirical results indicate that current state-of-the-art LLM agents experience significant performance degradation in multilingual settings, suggesting compounding effects across reasoning and execution steps.

Key takeaway

For AI Scientists and Machine Learning Engineers developing LLM agents for global applications, you must prioritize multilingual capabilities beyond simple translation. Your current state-of-the-art agents will likely suffer significant performance drops in complex, long-horizon multilingual workflows. Integrate robust multilingual evaluation early in your development cycle, focusing on how language variation impacts iterative reasoning and tool use to mitigate compounding errors.

Key insights

Multilinguality significantly degrades LLM agent performance in long-horizon tasks, a critical gap in current benchmarks.

Principles

• Multilinguality compounds effects across reasoning and execution steps.
• Jointly model language variation and procedural decision-making.

Method

A hybrid evaluation framework combines structural grading, executable verification, and LLM-based semantic assessment to capture functional correctness and linguistic consistency.

In practice

• Evaluate agents on heterogeneous multilingual inputs.
• Assess iterative reasoning and tool invocation in diverse languages.
• Verify structured output generation across language barriers.

Topics

• LLM Agents
• Multilingual Benchmarking
• Long-Horizon Tasks
• Workflow Automation
• Agent Evaluation
• Natural Language Processing

Best for: AI Engineer, Research Scientist, AI Product Manager, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual, multicultural, and grounded reasoning on real-world tasks, revealing significant performance gaps.
• Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows — BenchAgent reveals multi-agent LLM workflows often underperform single-agent setups under controlled conditions, except for specific runtime-generated designs.
• MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights — Multilingual LLM evaluation requires fine-grained, agentic diagnosis to transform scores into actionable insights, addressing cultural and deployment realities.
• Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages — Multi-LCB extends code generation benchmarks beyond Python to expose LLM multilingual performance disparities and overfitting.
• Benchmarking Open-Ended Multi-Agent Coordination in Language Agents — Open-ended multi-agent coordination is a distinct bottleneck for frontier LLMs, measurable by the $alem$ benchmark.
• Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions — Multi-Legal-Bench reveals LLM legal reasoning varies significantly across jurisdictions and tasks, with transfer quality tied to label-set alignment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.