Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

2026-05-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

A large-scale study evaluated Uncertainty Estimation (UE) methods across 22 languages, including high-, mid-, and low-resource settings, using two human-curated Q&A datasets. The research compared nine open-box and closed-box UE methods across various model sizes and architectures, including Claude 4.5 Sonnet, five Gemma3 models (0.27B, 1B, 4B, 12B, 27B), and three Qwen3 models (4B, 30B-A3B, 235B-A22B), totaling 44,000 language-specific instances. Key findings indicate that prompting models to reason in English for low-resource languages significantly improves UE performance, suggesting generation, not comprehension, is the bottleneck. This English reasoning also closes the UE performance gap between low and high-resource languages. Furthermore, the optimal UE method depends on model scale: open-box probability-based methods excel at smaller scales, while closed-box self-verbalized uncertainty becomes superior at larger scales.

Key takeaway

For Machine Learning Engineers deploying LLMs in multilingual environments, you should prioritize English reasoning for low-resource languages to improve uncertainty estimation and task accuracy. Consider open-box probability methods like "Token Entropy" for smaller models (e.g., Qwen3-4B) and self-verbalized uncertainty for larger models (e.g., Qwen3-235B) to optimize reliability. A global threshold for selective prediction simplifies deployment across diverse language settings.

Key insights

LLM uncertainty estimation in multilingual settings is significantly impacted by reasoning language and model scale.

Principles

• LLM generation, not comprehension, bottlenecks low-resource language UE.
• English reasoning improves UE for low-resource languages.
• UE method efficacy depends on model scale.

Method

Evaluated 9 UE methods on 2 human-curated MCQA datasets across 22 languages, eliciting long-form reasoning and using exact matching for correctness.

In practice

• Use English reasoning for low-resource language LLM tasks.
• Employ open-box UE for smaller LLMs (e.g., Gemma3-4B).
• Prefer self-verbalized UE for larger LLMs (e.g., Qwen3-235B).

Topics

• Uncertainty Estimation
• Multilingual LLMs
• Cross-lingual Reasoning
• Model Scaling
• Selective Prediction
• Open-box Methods
• Closed-box Methods

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models — A new taxonomy and evaluation framework reveal that consensus-based methods excel in quantifying LLM uncertainty, which decreases with model scale.
• LLM Parameters for Math Across Languages: Shared or Separate? — Multilingual LLMs exhibit partial, layer-dependent parameter overlap for math, with English dominating.
• A better method for identifying overconfident large language models — Cross-model disagreement combined with self-consistency more reliably identifies LLM overconfidence and hallucinations.
• BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation — LLMs struggle to consistently translate improved latent inference into rational downstream predictions in multi-turn Bayesian tasks.
• Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty — Structural uncertainty quantifies LLM logical reasoning consistency by measuring the stability of self-preference-induced rankings over sampled solutions.
• IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation — IUQ quantifies uncertainty in long-form LLM outputs using inter-sample consistency and intra-sample faithfulness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.