Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR

2026-05-28 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

IndiKLAR, an Indic extension of the KLAR-CLC benchmark, evaluates cross-lingual knowledge consistency in large language models for 18 of 22 scheduled Indian languages and 11 code-mixed variants. This benchmark reveals a significant knowledge recall gap, up to ~0.50, between native Indian languages and English. However, code-mixed inputs remarkably close most of this gap, bringing performance within ~0.05 of English without model-level intervention. The study also explores prompting strategies, including two-stage translate-then-answer, one-stage joint translation-and-answer, and Translate-in-Thought (TinT). A consistent "flip point" between incorrect and correct prediction is identified, lying between native and code-mixed settings, regardless of input form or internal model conversion.

Key takeaway

For NLP engineers deploying large language models in multilingual contexts, particularly with Indian languages, you should prioritize integrating code-mixed inputs. This approach can drastically reduce the cross-lingual knowledge recall gap from ~0.50 to ~0.05 compared to English, often without requiring model fine-tuning. Consider implementing advanced prompting strategies like Translate-in-Thought (TinT) to further enhance performance and consistency in low-resource language applications.

Key insights

Code-mixing inputs significantly improves large language model knowledge recall in Indian languages, nearly matching English performance.

Principles

• Code-mixing reduces cross-lingual knowledge consistency gaps.
• A consistent "flip point" for correct prediction exists between native and code-mixed inputs.

Method

The study evaluates prompting strategies like two-stage translate-then-answer, one-stage joint translation-and-answer, and Translate-in-Thought (TinT) for language conversion.

In practice

• Utilize code-mixed inputs to enhance LLM performance in Indian languages.
• Implement Translate-in-Thought (TinT) for improved cross-lingual knowledge recall.

Topics

• Large Language Models
• Cross-lingual Consistency
• Indian Languages
• Code-Mixing
• Prompting Strategies
• IndiKLAR Benchmark

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages — ILR-informed expert judgment reveals interpretable, domain-dependent cross-lingual variation in LLM outputs.
• Does Self-Consistency Improve the Recall of Encyclopedic Knowledge? — Self-consistency improves both symbolic reasoning and encyclopedic knowledge recall, even when CoT primarily aids reasoning.
• Merging Methods for Multilingual Knowledge Editing for Large Language Models: An Empirical Odyssey — Shared covariance in vector merging is crucial for effective multilingual knowledge editing in LLMs.
• Cross-Lingual Consensus: Aligning Multilingual Cultural Knowledge via Multilingual Self-Consistency — LLMs' latent cultural knowledge in local languages can be surfaced and transferred to improve English query performance and reduce Western bias.
• University of Tartu thesis: transfer learning boosts Estonian AI models - ERR — Cross-lingual transfer learning enables effective AI model development for low-resource languages despite limited data.
• Closing the Gap at CRAC 2026: Two-Stage Adaptation for LLM-Based Multilingual Coreference Resolution — Two-stage fine-tuning and optimized input formats enable LLMs to achieve state-of-the-art multilingual coreference resolution.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.