Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili

2026-06-26 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study investigated cross-lingual bias in large language models, specifically GPT-5.2 and Gemini 2.5 Flash, by comparing their responses to 4,900 symmetric English–Swahili prompt pairs across nine demographic bias axes. This yielded 19,600 completions, which were evaluated for stereotype prevalence, sentiment, refusal behavior, and cross-lingual semantic similarity. Findings indicate that bias transforms rather than transfers across languages; stereotype rates shifted by up to 12 percentage points on specific axes, Gemini's neutral-sentiment rate doubled in Swahili, and GPT-5.2 refused 169 English prompts but zero in Swahili, suggesting safety mechanisms are functionally anchored to English-language tokens. Over 55% of prompt pairs produced semantically dissimilar completions, reinforcing that English-only bias audits are insufficient for multilingual LLM deployment.

Key takeaway

For NLP Engineers deploying large language models in multilingual contexts, you must move beyond English-centric bias evaluations. Your current safety alignment and bias audits likely miss critical issues, as biases transform across languages, and refusal behaviors can be language-specific. Implement comprehensive cross-lingual testing, particularly for non-English languages like Swahili, to ensure equitable and safe model performance globally.

Key insights

Social biases in LLMs transform, not transfer, across languages, making English-only audits insufficient for multilingual deployment.

Principles

• LLM social biases transform, not transfer, across languages.
• English-only bias audits are inadequate for multilingual coverage.
• Safety mechanisms can be functionally anchored to specific language tokens.

Method

The study submitted 4,900 symmetric English–Swahili prompt pairs to GPT-5.2 and Gemini 2.5 Flash, evaluating 19,600 completions for stereotype prevalence, sentiment, refusal behavior, and cross-lingual semantic similarity.

In practice

• Evaluate LLM bias across multiple languages, not just English.
• Test safety mechanisms for language-specific refusal behaviors.
• Assess semantic similarity of cross-lingual outputs.

Topics

• Cross-Lingual Bias
• Large Language Models
• Bias Evaluation
• Multilingual AI Safety
• GPT-5.2
• Gemini 2.5 Flash

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs — Multilingual LLMs exhibit implicit US-centric and population-driven biases, exacerbated by instruction tuning.
• Why Do Safety Guardrails Degrade Across Languages? — A new IRT framework precisely diagnoses multilingual LLM safety failures by decoupling contributing factors beyond aggregate metrics.
• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• A Comparison of Methods to Bias Translation Toward Portuguese Variants — Biasing MT towards minority language variants is achievable through various lifecycle interventions.
• RedVox: Safety and Fairness Gaps in Speech Models Across Languages — Speech models exhibit significant safety and fairness vulnerabilities, especially in non-English languages and with spoken input.
• Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis — Direct translation of safety benchmarks fails to capture cultural context, underestimating LLM risks in multilingual settings.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.