Multilingual Disparities in LLM-Based Safety Judgments: Evidence from Brand Safety Applications

2026-06-26 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

Multilingual LLMs are increasingly employed as context-aware judges in real-world information systems, operating under the assumption that equivalent content receives consistent judgments across languages. This study investigates this assumption within brand safety applications, a global domain where automated ratings impact advertisers' reputations, publishers' revenues, and users' access to news. Researchers constructed a benchmark of LLM-generated safety ratings for 10,467 semantically aligned news articles across 13 languages. They discovered systematic cross-lingual disagreement in over 96% of cases where at least one language received a non-zero risk rating. Suitability ratings varied significantly by language. The main model rated English, German, and French content more strictly, while Polish, Hungarian, Greek, Turkish, and Persian content was rated more leniently. Robustness checks with two additional LLMs confirmed persistent language effects, though directional patterns varied by model, indicating unequal outcomes for semantically equivalent content.

Key takeaway

For NLP Engineers deploying multilingual LLMs for brand safety or content moderation, you must validate judgment consistency across all target languages. Your models may exhibit systematic biases, rating content in English, German, or French more strictly than Polish, Hungarian, Greek, Turkish, or Persian. Implement language-specific calibration and continuous monitoring to ensure equitable outcomes and prevent unintended impacts on publishers and users.

Key insights

Multilingual LLMs exhibit systematic cross-lingual disparities in safety judgments for semantically equivalent content, leading to unequal outcomes.

Principles

• Multilingual LLM safety judgments are not language-agnostic.
• Strictness of LLM safety ratings varies by language.
• Equivalent content can receive unequal LLM judgments.

Method

Constructed a benchmark of 10,467 semantically aligned news articles across 13 languages. Used LLMs to generate safety ratings and analyzed cross-lingual disagreement and suitability rating differences.

In practice

• Evaluate LLM safety judgments across all target languages.
• Calibrate LLM strictness per language for fairness.
• Monitor for disparate impact on content and revenue.

Topics

• Multilingual LLMs
• Brand Safety
• Content Moderation
• Cross-lingual Bias
• Safety Judgments
• Language Disparities

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Why Do Safety Guardrails Degrade Across Languages? — A new IRT framework precisely diagnoses multilingual LLM safety failures by decoupling contributing factors beyond aggregate metrics.
• Extending the Contact Hypothesis: Cross-Linguistic Evaluation of Religion and Nationality Bias When Prompting LLMs in German and Icelandic — LLMs exhibit religion and nationality biases, requiring cross-linguistic evaluation for fairness.
• Lost in Translation: Do LVLM Judges Generalize Across Languages? — LVLM judge models exhibit substantial cross-lingual performance variance, challenging current evaluation practices.
• Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis — Direct translation of safety benchmarks fails to capture cultural context, underestimating LLM risks in multilingual settings.
• Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignment — A plug-and-play MLC loss improves multilingual LLM safety alignment using only prompt variants, reducing resource needs.
• Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMs — Incidental multilingualism in LLMs leads to brittle, unequal, and opaque cross-lingual performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.