LLM Parameters for Math Across Languages: Shared or Separate?

2026-01-26 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

This study investigates whether mathematical reasoning parameters in multilingual Large Language Models (LLMs) are shared across languages or language-specific. Researchers analyzed Llama 1B, Qwen3 4B, and Llama 8B models across English, German, French, and Hindi using the MathNeurosurgery framework and Jaccard similarity. They found that math-associated parameters exhibit partial cross-lingual overlap, predominantly in intermediate model layers. English consistently showed the largest set of math-relevant parameters, correlating with its stronger reasoning performance, while lower-resource languages had smaller sets. The findings suggest that math capabilities are neither fully language-invariant nor entirely language-specific, but rather a blend with systematic language-dependent differences. Intervention experiments confirmed these parameters' collective influence, with scaling primarily correcting arithmetic errors.

Key takeaway

For AI Scientists and Machine Learning Engineers developing multilingual LLMs, understanding the language-dependent nature of mathematical reasoning parameters is crucial. You should consider that English-centric models may not generalize efficiently to other languages, especially those with different scripts like Hindi. Focus on optimizing intermediate model layers for cross-lingual math capabilities and explore targeted parameter interventions, such as pruning, to refine output formatting and in-context learning for specific language tasks.

Key insights

Multilingual LLMs exhibit partial, layer-dependent parameter overlap for math, with English dominating.

Principles

• Math-specific parameters show partial cross-lingual overlap.
• Overlap is strongest in intermediate LLM layers.
• English-centric pathways often dominate multilingual reasoning.

Method

Identify math-specific parameters using the MathNeurosurgery framework, comparing weight-activation products on math vs. non-math datasets. Measure cross-lingual overlap with the Jaccard coefficient.

In practice

• Focus optimization efforts on intermediate layers for multilingual math.
• Consider language-specific parameter tuning for lower-resource languages.
• Investigate pruning for output formatting improvements in specific tasks.

Topics

• Multilingual LLMs
• Mathematical Reasoning
• Parameter Localization
• Cross-lingual Transfer
• Model Interpretability
• Weight Pruning
• Jaccard Similarity

Code references

• luisavictor/math-across-languages

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• LLM Parameters for Math Across Languages: Shared or Separate? — LLM mathematical reasoning involves partially shared and partially language-specific parameters, with English dominating parameter sets.
• English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training — Multilingual post-training significantly improves LLM performance across languages and tasks, outperforming English-only approaches.
• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs reason effectively across 43 languages, with machine translation not degrading accuracy.
• Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR — Code-mixing inputs significantly improves large language model knowledge recall in Indian languages, nearly matching English performance.
• AI’s fluency in other languages hides a Western worldview that can mislead users − a scholar of Indonesian society explains — LLMs exhibit "epistemological persistence," retaining a Western worldview despite multilingual fluency due to English-centric training data.
• MATH-PT: A Math Reasoning Benchmark for European and Brazilian Portuguese — Math-PT is the first native Portuguese benchmark for LLM mathematical reasoning, revealing performance gaps in open-ended and visual problems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.