Culturally Grounded Image Captioning in Indigenous Languages with Vision-Language Models: Cascaded and Single-Stage Approaches

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Computer Vision · Depth: Expert, quick

Summary

The AmericasNLP 2026 shared task submission explores culturally grounded image captioning for under-resourced Indigenous languages, specifically Bribri, Guaraní, Yucatec Maya, Wixárika, and Orizaba Nahuatl. The research evaluates two architectural paradigms: a cascaded system and a single-stage approach. The cascaded system integrates a large vision-language model with a machine translation pipeline, demonstrating that culturally contextualized, persona-based prompting enhances performance over the official baseline in most comparable settings. The single-stage approach adapts PaliGemma 2 through LoRA fine-tuning, continued pre-training, and multilingual joint training. While relying on synthetic data and facing domain mismatch, multilingual training and continued pre-training improved automatic chrF++ scores compared to single-language LoRA fine-tuning in some scenarios. Currently, cascaded pipelines offer the strongest performance given existing data constraints, though single-stage models show promise for direct Indigenous-language image captioning despite their data limitations.

Key takeaway

For NLP Engineers developing solutions for under-resourced Indigenous languages, you should prioritize cascaded vision-language model architectures combined with machine translation pipelines. This approach, especially when incorporating culturally contextualized, persona-based prompting, currently yields stronger results than direct single-stage models like fine-tuned PaliGemma 2, given severe data scarcity. While single-stage methods are promising, their current reliance on synthetic data and domain mismatch means you should view them as a longer-term research path rather than an immediate deployment strategy.

Key insights

Culturally grounded image captioning for under-resourced Indigenous languages benefits most from cascaded VLMs with MT, though direct single-stage models show future promise.

Principles

• Culturally contextualized prompting improves VLM performance.
• Multilingual training can mitigate data scarcity.
• Cascaded systems currently outperform direct single-stage models.

Method

The cascaded method combines a VLM with a machine translation pipeline, using persona-based prompting. The single-stage method adapts PaliGemma 2 via LoRA fine-tuning, continued pre-training, and multilingual joint training.

In practice

• Implement persona-based prompting for cultural context.
• Explore multilingual joint training for low-resource languages.
• Consider cascaded VLM-MT for immediate results.

Topics

• Image Captioning
• Indigenous Languages
• Vision-Language Models
• Machine Translation
• Low-Resource NLP
• PaliGemma 2
• LoRA Fine-tuning

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages — Multimodal LLMs must integrate text, speech, and vision for low-resource languages, moving beyond English-centric benchmarks.
• Analysis of Machine Translators on Sentences Generated by Portuguese Image Captioning Models — Machine translation quality directly impacts image captioning model performance when adapting to new languages.
• Multilingual Reasoning Cascades Need More Context — Translation cascades benefit significantly from preserving original language context throughout the pipeline.
• GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models — Naively using VLM-generated captions can degrade accuracy due to anchoring effects and asymmetric error types.
• LLMs Infer Cultural Context but Fail to Apply It When Responding — LLMs infer cultural context but often fail to apply it in responses without explicit sequential prompting, revealing cultural biases.
• Zero-Shot Captioning for Cultural Heritage: Automated Image Analysis of Traditional Indonesian Clothing — Retrieval-augmented vision-language models effectively generate culturally accurate captions for low-resource heritage data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.