Borrowed Words, Borrowed Minds: Probing LLM Choice of English-Derived Loanwords in Japanese

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A new controlled evaluation dataset has been introduced to probe how large language models encode sociolinguistic variation in Japanese, specifically the choice between English-derived loanwords (gairaigo) and native Japanese equivalents. This dataset features 113 interchangeable lexical pairs embedded across six communicative contexts, encompassing formal and informal, spoken and written registers. Researchers evaluated 16 Japanese-capable LLMs using three tasks: sentence rating, pairwise choice, and masked word prediction. While both lexical forms were generally rated as natural, the models exhibited substantial divergence in contextual sensitivity and lexical preference. These findings highlight architectural differences in how socially grounded lexical alternatives are represented, suggesting that surface fluency in LLMs may mask instability in modeling pragmatic variation, which has implications for socially aware language generation and evaluation.

Key takeaway

For NLP Engineers developing Japanese LLMs, you must move beyond surface fluency metrics to evaluate true sociolinguistic competence. Your models' divergent contextual sensitivity and lexical preferences for gairaigo versus native equivalents indicate a need for more robust pragmatic modeling. Implement evaluation datasets that specifically probe socially meaningful lexical choices to ensure your LLMs generate contextually appropriate and culturally nuanced Japanese.

Key insights

LLMs exhibit instability in modeling sociolinguistic variation, particularly with Japanese loanwords, despite appearing fluent.

Principles

• Surface fluency can mask pragmatic modeling instability.
• Sociolinguistic choices reveal architectural LLM differences.

Method

A controlled dataset of 113 interchangeable lexical pairs across six contexts was used to evaluate 16 LLMs via sentence rating, pairwise choice, and masked word prediction tasks.

In practice

• Integrate sociolinguistic context into LLM evaluations.
• Develop LLMs for socially aware language generation.

Topics

• Large Language Models
• Japanese Language
• Sociolinguistics
• Lexical Choice
• Pragmatic Evaluation
• Language Generation

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• AI’s fluency in other languages hides a Western worldview that can mislead users − a scholar of Indonesian society explains — LLMs exhibit "epistemological persistence," retaining a Western worldview despite multilingual fluency due to English-centric training data.
• Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responses — LLMs inconsistently adjust response complexity, failing to reliably differentiate jargon and information across user-controlled levels.
• Multilinguality of Large Language Models From a Structural Perspective — LLMs' multilinguality involves structural differences, with low-resource languages diverging more from English.
• Hybrid Human-LLM Corpus Construction and LLM Evaluation for the Caused-Motion Construction — LLMs struggle with complex linguistic constructions like CMC, necessitating novel corpus creation methods.
• Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses — LLMs significantly struggle to infer pragmatic meaning from non-verbal communication, unlike their performance with verbal cues.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.