Translation Is Not Representation: English-Hub Routing in Cross-Lingual Bias Benchmarks

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Ethics & Bias · Depth: Expert, quick

Summary

A study on cross-lingual bias benchmarks, including JBBQ and KoBBQ, challenges the assumption that translated English bias probes measure identical constructs across languages. Researchers used 13B-parameter models, like Llama 2, Swallow, koen, and LLM-jp, matched on architecture but with varying language-training regimes. A multi-anchor logit lens revealed that English-centric models process Japanese and Korean inputs predominantly through English-script predictions in middle layers, even when Centered Kernel Alignment (CKA) between languages is high. Target-language adaptation reduced this English-script mass from 0.77 to 0.56 for Japanese (Swallow) and 0.78 to 0.71 for Korean (koen), with balanced bilingual pretraining (LLM-jp) lowering it to 0.19. Behaviorally, models showed more stereotype bias in English than Japanese (0.13 to 0.14 gaps), but Korean bias was weak and disappeared after adaptation. The findings indicate cross-lingual bias scores reflect genuine language-specific behavior, not an English-pivot artifact, despite non-comparable underlying representations. A four-step audit protocol for translated bias benchmarks is proposed.

Key takeaway

For NLP Engineers evaluating cross-lingual models for bias, recognize that translated probes may not measure identical constructs across languages. Your models might process non-English inputs via an English-hub, even with high geometric convergence. Apply the proposed four-step audit protocol to ensure bias scores genuinely reflect language-specific behavior, and consider balanced bilingual pretraining or targeted adaptation to mitigate English-centric routing.

Key insights

Cross-lingual bias scores reflect language-specific behavior despite English-hub routing and non-comparable underlying representations.

Principles

• Geometric convergence can mask English-hub routing in cross-lingual models.
• Target-language adaptation reduces English-script processing mass.
• Behavioral bias can be language-specific, independent of representation.

Method

A four-step audit protocol for translated bias benchmarks is proposed to address the dissociation between representation and behavior.

In practice

• Audit translated bias benchmarks using the proposed four-step protocol.
• Consider language-specific adaptation to reduce English-hub routing.

Topics

• Cross-lingual Bias
• Large Language Models
• English-Hub Routing
• Model Representation
• Language Adaptation
• Bias Benchmarks

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs — Multilingual LLMs exhibit implicit US-centric and population-driven biases, exacerbated by instruction tuning.
• RBCorr: Response Bias Correction in Language Models — RBCorr effectively mitigates language model response bias and improves performance using a simple, low-cost LogProbs-based calibration method.
• Extending the Contact Hypothesis: Cross-Linguistic Evaluation of Religion and Nationality Bias When Prompting LLMs in German and Icelandic — LLMs exhibit religion and nationality biases, requiring cross-linguistic evaluation for fairness.
• CLP-Transfer: Cross-Lingual and Progressive Transfer Learning — CLP-Transfer simplifies cross-lingual model transfer using token overlap and a small helper model, but shows limited downstream task performance.
• How Dialect Variation Challenges Natural Language Processing Systems — NLP systems struggle with dialects due to biased training data, leading to technical and social inequities.
• Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs — Multilingual LLMs exhibit implicit US-centric and demographic biases when answering locale-ambiguous questions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.