PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

PluraMath is a new dataset designed to extend mathematical reasoning evaluation for Large Language Models (LLMs) beyond high-resource languages. It builds upon the PolyMath dataset by adding 18 underrepresented languages across 6 language families, ranging from mid-resource to extreme low-resource settings. The dataset was constructed using a human-curated pipeline where native speakers validated pre-computed translations. Researchers utilized PluraMath to benchmark 27 reasoning LLMs, encompassing small, mid-size, large, and closed-source ensemble models, to assess the multilingual mathematical reasoning capabilities of leading models. Analysis revealed a persistent performance gap in mathematical reasoning between high-resource and underrepresented languages, with better instruction-following ability correlating with stronger results. The dataset, its acquisition pipeline, and evaluation framework are fully open-sourced to facilitate multilingual benchmark development.

Key takeaway

For Machine Learning Engineers developing multilingual LLMs, PluraMath highlights critical performance disparities in mathematical reasoning across languages. You should prioritize evaluating your models on underrepresented language benchmarks like PluraMath to identify and address these gaps. Focus on improving instruction-following capabilities, as this directly correlates with enhanced reasoning performance in diverse linguistic settings, ensuring more equitable global model utility.

Key insights

Mathematical reasoning in LLMs shows a persistent performance gap in underrepresented languages, despite new multilingual benchmarks.

Principles

• LLM mathematical reasoning varies significantly by language resource level.
• Instruction-following ability correlates with better multilingual reasoning.
• Human validation is crucial for low-resource language dataset quality.

Method

PluraMath was constructed via a human-curated pipeline, involving native speakers validating pre-computed translations for 18 underrepresented languages.

In practice

• Benchmark LLMs on PluraMath for multilingual reasoning gaps.
• Use human-curated validation for low-resource NLP datasets.
• Integrate instruction-following improvements for multilingual LLMs.

Topics

• PluraMath Dataset
• Multilingual LLMs
• Mathematical Reasoning
• Low-Resource Languages
• LLM Benchmarking
• Instruction Following

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• MATH-PT: A Math Reasoning Benchmark for European and Brazilian Portuguese — Math-PT is the first native Portuguese benchmark for LLM mathematical reasoning, revealing performance gaps in open-ended and visual problems.
• Semantic adapters in text-to-SQL for low-resource languages: the importance of semantic information — Injecting semantic structural knowledge into LLMs significantly boosts Text-to-SQL performance for low-resource languages.
• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual and multicultural reasoning in grounded, real-world tasks.
• PyraMathBench: Evaluating and Improving Mathematical Capability in Large Language Models — LLMs' mathematical failures stem from poor numerical computation and abstract numerical handling, necessitating integrated benchmarks and specialized modules.
• Why LLMs Are Worse at Math in Other Languages — And It’s Not Just the Data — LLM math performance differences across languages are linked to varying allocations and overlaps of "math parameters" within the model's architecture.
• Why isn’t LLM reasoning done in vector space instead of natural language?[D] — LLMs internally use vectors, but natural language reasoning offers crucial interpretability and debuggability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.