Stereotyped by Silence: How LLMs Erase Northeast Indian Languages Through Omission and Orthographic Corruption

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A study reveals that large language models (LLMs) systematically erase Northeast Indian languages through omission and orthographic corruption, perpetuating cultural stereotypes. Researchers found two primary failure modes: entity-level invisibility, where state-of-the-art NER systems scored F1=0.000 on critical entities like Khasi surnames and Garo festivals; and orthographic corruption, with LLM tokenizers corrupting diacritics (ï, ñ) and the Garo morpheme boundary marker (U+00B7) at rates of 18.8–50% across four of five evaluated models. To address this, the study developed NortheastNER, an NER system achieving F1=0.964 on six entity categories using XLM-RoBERTa-base, and a custom multilingual tokenizer that reduced tokens by 26–50% compared to five baseline LLMs. These findings highlight stereotype-by-omission as a distinct harm, advocating for cultural representation audits in multilingual NLP evaluation.

Key takeaway

For NLP Engineers developing or deploying multilingual LLMs, you must prioritize cultural representation audits to prevent systematic erasure of underrepresented languages. Your current tokenizers and NER systems likely fail to recognize critical entities and corrupt orthography, with F1=0.000 scores and 18.8–50% corruption rates observed. Consider integrating specialized tools like NortheastNER or custom multilingual tokenizers to ensure equitable language support and avoid perpetuating harmful stereotypes.

Key insights

LLMs perpetuate cultural stereotypes by systematically omitting and corrupting underrepresented languages, causing measurable harm.

Principles

• LLM biases extend beyond associations to omission.
• Orthographic corruption impacts semantic meaning.
• Cultural representation audits are crucial for NLP.

Method

The study involved empirical evidence of NER system failures (F1=0.000) and tokenizer corruption (18.8–50%) for Northeast Indian languages, then developed NortheastNER (F1=0.964) and a custom multilingual tokenizer for remediation.

In practice

• Implement cultural representation audits.
• Develop custom multilingual tokenizers.
• Use specialized NER for underrepresented languages.

Topics

• Large Language Models
• Multilingual NLP
• Northeast Indian Languages
• NER Systems
• Tokenizer Bias
• Cultural Representation Audits

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Ethicist

Related on AIssential

• Linguistic Issues in Computational Linguistics for Non-English Languages — English-centric LLM design creates systematic linguistic weaknesses for non-English languages across multiple levels.
• Twenty Years of HAREM: A Reproducible Audit and Reassessment of Portuguese Named Entity Recognition — Data contamination in benchmarks significantly distorts model performance and masks true generalization capabilities.
• Disparities In Negation Understanding Across Languages In Vision-Language Models — VLMs exhibit affirmation bias, with negation understanding varying significantly across typologically diverse languages.
• The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs — LLMs often access local cultural knowledge better in local languages, despite lower raw accuracy due to proficiency gaps.
• How Dialect Variation Challenges Natural Language Processing Systems — NLP systems struggle with dialects due to biased training data, leading to technical and social inequities.
• Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models — Cross-lingual fairness and tokenization efficiency are not fundamentally at odds in multilingual LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.