The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs

2026-06-05 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A study titled "The Masked Advantage" investigates how large language models (LLMs) access cultural knowledge across different languages. Researchers developed a controlled framework using real-world cultural questions from 13 locales and roughly 80 models. They crossed question types (culture-agnostic vs. culture-specific) with query languages (English vs. local language), employing a 1PL item response theory model to separate language proficiency from knowledge access. Findings indicate an English advantage for culture-agnostic questions due to stronger proficiency. However, after adjusting for this proficiency gap, local languages consistently showed a positive knowledge-access advantage in nearly all settings, often masked by raw accuracy. This suggests local cultural knowledge is more accessible via local languages, despite potential proficiency limitations.

Key takeaway

For NLP Engineers evaluating large language models for culturally grounded applications, do not solely rely on raw accuracy metrics, especially when assessing local language performance. Your evaluations should account for language proficiency gaps, as local languages often provide a superior knowledge-access advantage for cultural content, even if raw scores appear lower. Consider using a framework that separates proficiency from knowledge access to reveal true cultural understanding.

Key insights

Local languages offer a knowledge-access advantage for cultural questions in LLMs, often masked by English proficiency.

Principles

• Raw accuracy can mask true knowledge access.
• Separate language proficiency from knowledge access.
• Local languages enhance cultural knowledge retrieval.

Method

A controlled framework crosses question type (culture-agnostic/specific) with query language (English/local), using a 1PL item response theory model to separate proficiency from knowledge access.

In practice

• Evaluate LLMs with real-world cultural questions.
• Test both English and local language queries.
• Consider regionally aligned LLMs for cultural tasks.

Topics

• Large Language Models
• Cultural Knowledge
• Multilingual LLMs
• Language Proficiency
• LLM Evaluation
• Item Response Theory

Best for: Research Scientist, AI Engineer, Machine Learning Engineer, AI Scientist, NLP Engineer

Related on AIssential

• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual, multicultural, and grounded reasoning on real-world tasks, revealing significant performance gaps.
• The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs — LLMs often access local cultural knowledge better in local languages, despite lower raw accuracy due to proficiency gaps.
• CARTE: A Benchmark for Mapping Language Model Knowledge Across France — LLMs exhibit significant knowledge gaps in fine-grained, geographically specific regional understanding, particularly within countries.
• Cross-Lingual Consensus: Aligning Multilingual Cultural Knowledge via Multilingual Self-Consistency — LLMs' latent cultural knowledge in local languages can be surfaced and transferred to improve English query performance and reduce Western bias.
• Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese — Chinese LLM safety requires culturally specific benchmarks to counter linguistic and cultural evasion techniques.
• AI’s fluency in other languages hides a Western worldview that can mislead users − a scholar of Indonesian society explains — LLMs exhibit "epistemological persistence," retaining a Western worldview despite multilingual fluency due to English-centric training data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.