Code-Switching Reveals Language Anchoring in Multilingual LLMs

2026-05-26 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

Multilingual Large Language Models (MLLMs) often degrade performance when processing Code-Switched (CS) inputs, where languages are mixed within an interaction. Researchers introduce Anchor Bias, a geometric measure quantifying whether a CS hidden state aligns closer to its source or target language counterpart. Across diverse MLLMs, including Aya-Expanse-8B, Qwen3.5-4B/9B/27B, Llama-3.1-70B-Instruct, and Mixtral-8x7B-Instruct, Anchor Bias reveals a consistent grammar-frame effect: source-framed CS stays source-anchored, while target-framed CS shifts target-ward, correlating with larger Question Answering (QA) degradation. To mitigate this, CANVAS (Contextual Anchor-based Neural Vector Alignment Steering) is proposed. This inference-time intervention extracts a source-side canvas and softly steers target-language hidden states toward it during prefill in the upper 30% of transformer layers. CANVAS consistently recovers QA F1 across MLLMs and CS conditions, showing an average F1 improvement of +0.73 and generalizing to multi-turn dialogs.

Key takeaway

For AI Scientists and Machine Learning Engineers developing or deploying multilingual LLMs, understanding and mitigating code-switching performance degradation is crucial. Your teams should consider integrating CANVAS, an inference-time intervention, to improve Question Answering F1 scores by aligning target-language hidden states toward a source canvas. This method, which adds only modest sub-second latency, offers a practical approach to enhance MLLM robustness in mixed-language environments, especially for target-framed inputs.

Key insights

Grammatical frame dictates multilingual LLM internal language anchoring, impacting code-switched performance.

Principles

• Grammatical frame strongly shapes internal language anchoring in MLLMs.
• Target-ward representation shifts correlate with larger QA performance drops.
• Source-canvas alignment can mitigate code-switching inference failures.

Method

CANVAS partitions CS input, constructs a source canvas from upper-layer hidden states, scores canvas alignment, and adaptively interpolates target-language hidden states toward the source canvas during prefill in the final 30% of layers.

In practice

• Use Anchor Bias to diagnose MLLM code-switching representation behavior.
• Apply CANVAS for inference-time mitigation of CS performance drops.
• Focus intervention on upper transformer layers for semantic alignment.

Topics

• Multilingual LLMs
• Code-Switching
• Language Anchoring
• CANVAS Intervention
• Representation Analysis
• Question Answering

Code references

• jeonghyunpark2002/CANVAS
• pemistahl/lingua-py

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Code-Switching Reveals Language Anchoring in Multilingual LLMs — Language anchoring in MLLMs, revealed by code-switching, causes performance degradation, which can be mitigated by steering hidden states.
• Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning — Multilingual code-switching instruction tuning improves LLM performance beyond bilingual settings.
• Linguistic Issues in Computational Linguistics for Non-English Languages — English-centric LLM design creates systematic linguistic weaknesses for non-English languages across multiple levels.
• MAAM: Anchor-Preserving Compression and Contextual Calibration for Chinese Discriminatory Language Detection — MAAM uses anchor preservation and contextual calibration for robust, compact Chinese discriminatory language detection.
• From Brewing to Resolution: Tracing the Internal Lifecycle of Code Reasoning in LLMs — LLMs' code reasoning involves an internal "brewing" phase before diverging into distinct resolution outcomes, revealing hidden failure modes.
• Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization — LLMs perform cross-lingual summarization and translation jointly in later layers, not sequentially.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.