Exploring Coherence of LLMs in Multilingual Question Answering

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A recent study investigates the coherence of Large Language Models (LLMs) in multilingual Question Answering (QA) tasks, addressing a gap in research predominantly focused on English. The work evaluates LLMs ranging from 3.8 billion to 235 billion parameters across six languages: English, Italian, German, Chinese, Japanese, and Vietnamese. Researchers aimed to understand how coherence, defined as the ability to produce consistent responses to semantically equivalent questions, scales with model capacity and varies across different languages. Key findings indicate that LLM coherence is not solely determined by model size or accuracy. Furthermore, the study reveals significant variations in coherence levels between languages for certain models, suggesting language-specific challenges or model biases in maintaining consistent outputs.

Key takeaway

For NLP Engineers developing multilingual QA systems, recognize that LLM coherence is not a direct function of model size or accuracy. You should specifically test for response consistency across languages, as performance can vary significantly beyond standard accuracy metrics. This necessitates dedicated evaluation strategies for multilingual coherence to ensure reliable and trustworthy AI applications.

Key insights

LLM coherence in multilingual QA is not solely tied to model size or accuracy, varying significantly across languages.

Principles

• LLM coherence is distinct from accuracy.
• Coherence varies by language for some models.
• Model size does not guarantee coherence.

Method

The study evaluated LLM coherence by testing consistent responses to semantically equivalent questions across six languages (English, Italian, German, Chinese, Japanese, Vietnamese) using models from 3.8B to 235B parameters.

Topics

• Large Language Models
• Multilingual QA
• LLM Coherence
• Model Evaluation
• Cross-lingual NLP
• Model Scaling

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs reason effectively across 43 languages, with machine translation not degrading accuracy.
• Building LMs that can model the world and model themselves | ARC Prize @ MIT — LLMs require explicit training for internal coherence, encompassing both factual consistency and accurate self-assessment of knowledge.
• Korean Culture into LLM Alignment: Toward Cultural Coherence — LLM cultural alignment requires constructive, locale-specific definitions of coherence, moving beyond mere output suppression.
• Cohere Launches Tiny Multilingual Open Weight Model — Tiny Aya offers open-weight, multilingual AI models to enhance linguistic diversity and edge device accessibility.
• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• AI’s fluency in other languages hides a Western worldview that can mislead users − a scholar of Indonesian society explains — LLMs exhibit "epistemological persistence," retaining a Western worldview despite multilingual fluency due to English-centric training data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.