mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?

2026-06-26 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

mmPISA-bench is introduced as a compact, high-quality multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment (PISA). It comprises 25 multiple-choice questions, each provided in official human translations across 43 languages and complemented by machine-translated versions, totaling 2,150 data points. Evaluating two mainstream proprietary LLMs, the study found that these models reason effectively across all languages, achieving accuracy comparable to human test-takers, despite some performance variations. Crucially, machine-translated questions did not degrade accuracy compared to human translations, suggesting synthetic data can be adequate for large-scale multilingual evaluations. The analysis also revealed that LLM usage in certain languages can be both more expensive and less accurate due to token usage and inference costs.

Key takeaway

For NLP Engineers developing or deploying multilingual LLMs, you should consider integrating high-quality machine translation for creating large-scale evaluation datasets, especially when official human translations are scarce. This approach can maintain evaluation accuracy while significantly expanding language coverage. Additionally, analyze token usage and inference costs per language to optimize deployment strategies, as performance and expense can vary considerably across different linguistic contexts.

Key insights

Modern LLMs demonstrate effective multilingual reasoning, even with machine-translated evaluation data.

Principles

• LLMs reason effectively across diverse languages.
• Machine translation can yield high-quality evaluation data.
• LLM performance and cost vary by language.

Method

mmPISA-bench uses 25 PISA-derived multiple-choice reasoning questions, translated into 43 human and machine versions, to evaluate LLMs across language, effort, and translation types.

In practice

• Utilize machine translation for multilingual evaluation data.
• Benchmark LLM cost-efficiency per language.

Topics

• Multilingual LLMs
• Reasoning Benchmarks
• PISA Assessment
• Machine Translation Quality
• LLM Evaluation
• Inference Cost

Best for: AI Engineer, CTO, VP of Engineering/Data, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs demonstrate effective multilingual reasoning across 43 languages, with machine translations proving as effective as human ones.
• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs reason effectively across 43 languages, with machine translation not degrading accuracy.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.
• Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions — Multi-Legal-Bench reveals LLM legal reasoning varies significantly across jurisdictions and tasks, with transfer quality tied to label-set alignment.
• A Multilingual Paradigm for AI Created by All, for All — Multilingual AI evaluations must be culturally nuanced, context-specific, and independently verified to ensure equitable global AI performance.
• Linguistic Issues in Computational Linguistics for Non-English Languages — English-centric LLM design creates systematic linguistic weaknesses for non-English languages across multiple levels.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.