PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

PolyWorkBench is a new benchmark designed to evaluate large language model (LLM) agents on multilingual long-horizon workplace workflows. It addresses the gap where most existing benchmarks assume monolingual settings, despite real-world applications often involving multilingual inputs and outputs. PolyWorkBench comprises 67 tasks spanning five domains: commerce, knowledge work, legal analysis, localization, and manufacturing, requiring agents to process diverse multilingual inputs, perform iterative reasoning, invoke external tools, and produce structured outputs. The benchmark employs a hybrid evaluation framework, integrating structural grading, executable verification, and LLM-based semantic assessment to ensure both functional correctness and linguistic consistency. Empirical results reveal that current state-of-the-art LLM agents experience significant performance degradation in multilingual workflow scenarios compared to their monolingual counterparts. This degradation highlights compounding effects of multilinguality across reasoning and execution.

Key takeaway

For machine learning engineers developing or deploying LLM agents for real-world, multilingual applications, you must account for significant performance degradation. Your current state-of-the-art agents will likely struggle with multilingual inputs and outputs across complex workflows. Prioritize research and development into jointly modeling language variation and procedural decision-making to build robust agents capable of handling diverse linguistic environments effectively.

Key insights

Multilinguality significantly degrades LLM agent performance in long-horizon tasks, introducing compounding effects across reasoning and execution.

Principles

• Multilinguality compounds effects across reasoning and execution steps.
• Jointly model language variation and procedural decision-making.

Method

PolyWorkBench proposes a hybrid evaluation framework combining structural grading, executable verification, and LLM-based semantic assessment to capture functional correctness and linguistic consistency.

In practice

• Evaluate LLM agents using multilingual long-horizon benchmarks.
• Assess agent performance across diverse domains like commerce and legal analysis.

Topics

• LLM Agents
• Multilingual NLP
• Benchmarking
• Long-Horizon Tasks
• Tool Use
• Workflow Automation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual, multicultural, and grounded reasoning on real-world tasks, revealing significant performance gaps.
• CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies — Active communication is crucial for LLM agents to succeed in long-horizon, multi-agent economic simulations.
• Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages — Multi-LCB extends code generation benchmarks beyond Python to expose LLM multilingual performance disparities and overfitting.
• Open-SWE-Traces: Advancing Dual-Mode Multilingual Distillation for Software Engineering Agents — Dual-mode multilingual distillation with diverse agentic trajectories significantly improves LLM performance in software engineering tasks.
• Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language — Automating executable visual workflow generation from natural language remains a significant challenge for current LLMs.
• MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights — Multilingual LLM evaluation requires fine-grained, agentic diagnosis to transform scores into actionable insights, addressing cultural and deployment realities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.