Evaluating Cross-Lingual Behavior and Consistency of Multimodal Large Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Computer Vision · Depth: Expert, quick

Summary

Two new benchmarks, KnowRecall and VisRecall, have been introduced to evaluate the cross-lingual behavior and consistency of multimodal large language models (MLLMs). This initiative addresses the significant challenge MLLMs face in maintaining consistent performance across various languages, particularly when integrating cultural knowledge. KnowRecall is a visual question answering benchmark designed to measure factual knowledge consistency across 15 languages, focusing on cultural and historical questions related to global landmarks. VisRecall, conversely, assesses visual memory consistency by prompting models to describe landmark appearances in 9 languages without image access. Experimental results indicate that even advanced MLLMs, including proprietary versions, currently struggle to achieve satisfactory cross-lingual consistency, highlighting a critical need for more robust, truly multilingual, and culturally aware model development approaches.

Key takeaway

For AI Scientists and NLP Engineers developing or deploying multimodal large language models for global applications, recognize that current advanced MLLMs exhibit significant cross-lingual and cultural consistency issues. You should prioritize integrating robust multilingual and culturally aware design principles into your model architectures and training data. This is crucial for ensuring reliable and equitable performance across diverse linguistic and cultural contexts, moving beyond mere language translation to true cultural understanding.

Key insights

New benchmarks, KnowRecall and VisRecall, reveal advanced MLLMs struggle with cross-lingual and cultural consistency, necessitating robust multilingual approaches.

Principles

• MLLMs require cultural knowledge.
• Cross-lingual consistency is challenging.
• Benchmarks drive MLLM improvement.

Method

The method involves two benchmarks: KnowRecall for VQA on cultural/historical facts across 15 languages, and VisRecall for assessing visual memory consistency by describing landmarks in 9 languages without images.

In practice

• Evaluate MLLMs with KnowRecall.
• Test visual memory via VisRecall.
• Prioritize cultural data in MLLM training.

Topics

• Multimodal Large Language Models
• Cross-lingual Evaluation
• Cultural Knowledge
• Visual Question Answering
• Benchmark Datasets
• Model Consistency

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Cross-Lingual Response Consistency in Large Language Models: An ILR-Informed Evaluation of Claude Across Six Languages — ILR-informed expert judgment reveals interpretable, domain-dependent cross-lingual variation in LLM outputs.
• Cross-Lingual Consensus: Aligning Multilingual Cultural Knowledge via Multilingual Self-Consistency — LLMs' latent cultural knowledge in local languages can be surfaced and transferred to improve English query performance and reduce Western bias.
• Grandes Modelos de Linguagem Multimodais (MLLMs): Da Teoria \`a Pr\'atica — MLLMs merge LLM language capabilities with multimodal perception, advancing AI's understanding of diverse data.
• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual and multicultural reasoning in grounded, real-world tasks.
• Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark — Monolingual LLMs for under-resourced languages require specific evaluation against multilingual counterparts.
• AI’s fluency in other languages hides a Western worldview that can mislead users − a scholar of Indonesian society explains — LLMs exhibit "epistemological persistence," retaining a Western worldview despite multilingual fluency due to English-centric training data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.