Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A large-scale study evaluated nine Uncertainty Estimation (UE) methods in Large Language Models across 22 languages, including high-, mid-, and low-resource settings. Using human-curated Q&A datasets, the research compared open and closed-box UE techniques across various model sizes and architectures, eliciting long-form reasoning to avoid evaluation noise. Key findings indicate that prompting models to reason in English for low-resource language questions substantially improves UE performance, suggesting generation, not comprehension, is the bottleneck. This English reasoning also closes the UE performance gap between low and high-resource languages. Furthermore, the optimal UE method depends on model scale: open-box probability-based methods excel at smaller scales, while closed-box self-verbalized uncertainty is superior for larger models. The study also provides guidance on threshold selection for selective prediction in multilingual contexts.

Key takeaway

For Machine Learning Engineers deploying multilingual LLMs, especially in low-resource contexts, prioritize prompting models to reason in English to significantly improve uncertainty estimation. This strategy addresses generation bottlenecks and closes performance gaps across languages. When selecting uncertainty estimation methods, align your choice with model scale: use probability-based approaches for smaller models and self-verbalized uncertainty for larger ones. Calibrate abstention thresholds carefully for robust selective prediction in diverse language environments.

Key insights

Prompting LLMs to reason in English improves multilingual uncertainty estimation, especially for low-resource languages.

Principles

• LLM generation, not comprehension, often bottlenecks low-resource language performance.
• Reasoning language significantly impacts multilingual UE performance.
• Optimal UE methods vary by LLM scale.

Method

Evaluated nine open and closed-box UE methods across 22 languages using human-curated Q&A datasets, eliciting long-form reasoning.

In practice

• Use English reasoning prompts for low-resource language questions.
• Select UE methods based on model scale (probability-based for small, self-verbalized for large).
• Calibrate abstention thresholds for multilingual selective prediction.

Topics

• Uncertainty Estimation
• Large Language Models
• Multilingual NLP
• Crosslingual Transfer
• Low-Resource Languages
• Selective Prediction

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models — A new taxonomy and evaluation framework reveal that consensus-based methods excel in quantifying LLM uncertainty, which decreases with model scale.
• LLM Parameters for Math Across Languages: Shared or Separate? — Multilingual LLMs exhibit partial, layer-dependent parameter overlap for math, with English dominating.
• A better method for identifying overconfident large language models — Cross-model disagreement combined with self-consistency more reliably identifies LLM overconfidence and hallucinations.
• BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation — LLMs struggle to consistently translate improved latent inference into rational downstream predictions in multi-turn Bayesian tasks.
• IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation — IUQ quantifies uncertainty in long-form LLM outputs using inter-sample consistency and intra-sample faithfulness.
• Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty — Structural uncertainty quantifies LLM logical reasoning consistency by measuring the stability of self-preference-induced rankings over sampled solutions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.