Why LLMs Are Worse at Math in Other Languages — And It’s Not Just the Data

2026-06-23 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A recent paper investigates why Large Language Models (LLMs) exhibit lower math accuracy in non-English languages, particularly low-resource ones, even when controlling for training data. Employing a mechanistic interpretability approach, similar to MathNeurosurgery, the research uses statistical signals from the forward pass to identify "math parameters" within models like Llama and Qwen. Findings indicate these math parameters partially overlap across languages, with the highest concentration of shared cross-lingual arithmetic modules in the model's middle layers. English consistently utilizes the largest set of math-related parameters, while low-resource languages allocate noticeably fewer. Weight intervention experiments confirmed these parameters are critical for math ability, suggesting performance gaps stem from parameter allocation, not solely data volume.

Key takeaway

For Machine Learning Engineers optimizing multilingual LLMs, understand that poor math performance in low-resource languages is not solely a data problem. Your models may not be allocating sufficient "math parameters" for these languages. Consider exploring methods beyond simply adding more data, such as targeted architectural interventions or parameter allocation strategies, to improve cross-lingual arithmetic capabilities, especially in the model's middle layers.

Key insights

LLM math performance differences across languages are linked to varying allocations and overlaps of "math parameters" within the model's architecture.

Principles

• LLM math parameters partially overlap across languages.
• Cross-lingual arithmetic modules concentrate in middle layers.
• English utilizes more math-related parameters than low-resource languages.

Method

Identify critical "math parameters" using statistical signals from the forward pass, then extract and compare them across languages in open-source models like Llama and Qwen.

In practice

• Investigate parameter allocation strategies for low-resource languages.
• Focus multilingual math improvements on middle-layer architectures.

Topics

• Large Language Models
• Multilingual AI
• Mechanistic Interpretability
• Math Reasoning
• Parameter Allocation
• Low-Resource Languages

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• Why LLMs Get Math Wrong — Large Language Models (LLMs) often make mistakes in complex math problems and misinterpret tables because they process all input as flat text token sequences, treating math as patterns to predict rath…
• English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training — Multilingual post-training significantly improves LLM performance across languages and tasks, outperforming English-only approaches.
• MATH-PT: A Math Reasoning Benchmark for European and Brazilian Portuguese — Math-PT is the first native Portuguese benchmark for LLM mathematical reasoning, revealing performance gaps in open-ended and visual problems.
• What are small language models and how do they differ from large ones? — SLMs offer efficient, cost-effective specialization, while LLMs provide versatile, sophisticated capabilities for complex tasks.
• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs reason effectively across 43 languages, with machine translation not degrading accuracy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.