MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs

2024-01-30 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

MELLA, a novel multimodal multilingual dataset, addresses the significant performance gap of Multimodal Large Language Models (MLLMs) in low-resource languages. Existing MLLMs often provide "thin descriptions" due to insufficient cultural groundedness and linguistic capability. This study introduces a dual-source data strategy, collecting 6.8 million image-text pairs across eight low-resource languages: Arabic, Czech, Hungarian, Korean, Russian, Serbian, Thai, and Vietnamese. MELLA comprises D_know (native web alt-text for cultural knowledge) and D_ling (MLLM-generated, translated captions for linguistic fluency). Experiments demonstrate that fine-tuning MLLM backbones like InternVL2-8B and QwenVL2-7B on MELLA significantly improves both cultural understanding (keyword accuracy) and linguistic skills (BLEU, ROUGE-L, METEOR), enabling "thick descriptions."

Key takeaway

For AI Scientists and Machine Learning Engineers expanding MLLM capabilities to low-resource languages, you should prioritize both linguistic fluency and cultural groundedness. Relying solely on machine translation for data will yield superficial outputs. Instead, adopt a dual-source data strategy, integrating native cultural context from sources like web alt-text with linguistically rich translated captions to build truly effective and inclusive multimodal AI systems.

Key insights

Effective low-resource MLLMs require both linguistic fluency and deep cultural understanding, beyond mere translation.

Principles

• Image meaning includes literal denotation and symbolic connotation.
• Translation-only data produces "thin descriptions" lacking cultural depth.
• Authentic, in-culture data is vital for cultural groundedness.

Method

A dual-source strategy collects native web alt-text for cultural knowledge (D_know) and MLLM-generated, translated captions for linguistic capability (D_ling), then combines them for unified fine-tuning.

In practice

• Source native web alt-text for cultural knowledge.
• Generate and translate MLLM captions for linguistic fluency.
• Combine diverse datasets for balanced MLLM enhancement.

Topics

• Multimodal LLMs
• Low-Resource Languages
• Cultural Groundedness
• Dataset Curation
• Supervised Fine-tuning
• Image Captioning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks — CulturALL benchmarks LLMs' multilingual and multicultural reasoning in grounded, real-world tasks.
• Culturally Grounded Image Captioning in Indigenous Languages with Vision-Language Models: Cascaded and Single-Stage Approaches — Culturally grounded image captioning for under-resourced Indigenous languages benefits most from cascaded VLMs with MT, though direct single-stage models show future promise.
• AI’s English Problem—and Why We Should Care — English dominance in AI training data creates a global language gap, reinforcing digital divides.
• Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research — LLMs offer transformative potential for low-resource language research in humanities, despite significant data and ethical challenges.
• Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation — Culturally aligned instruction-tuning with BLADE dataset significantly improves honorific consistency in multilingual Bangla LLM generation.
• SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses — Large Language Models often degrade in cross-lingual emotional support, particularly cultural grounding, when transitioning to low-resource languages like Ukrainian.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.