USP at AmericasNLP 2026 Shared Task: Culturally-Aware Image Captioning for Indigenous Languages via Vision-Language Models and Fine-Tuned Neural Machine Translation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

The USP system participated in the AmericasNLP 2026 Shared Task, focusing on Culturally Relevant Image Captioning for Indigenous Languages. It targeted Guaraní, Maya Yucateco, Nahuatl, Wixárika, and Bribri. The approach uses a two-stage cascade: Qwen3-VL-8B-Instruct generates Spanish captions via cultural prompts. Then, language-specific fine-tuned NLLB-200-distilled-600M models translate these into the target indigenous languages. Training utilized AmericasNLP 2023 data, augmented with public parallel corpora. The system achieved competitive results, including 3rd place in Guaraní human evaluation (2.41/5.0) and 5th in Bribri (1.09/5.0) among eight teams. A critical finding revealed NLLB-200-distilled-600M silently produces English output for Bribri and Maya Yucateco due to missing vocabulary entries.

Key takeaway

For NLP Engineers developing multilingual image captioning for indigenous languages, consider a cascaded VLM-NMT approach. Critically validate your NMT model's vocabulary. The NLLB-200-distilled-600M, for example, silently fails for languages like Bribri and Maya Yucateco, producing English output. This silent failure necessitates explicit vocabulary checks and robust error handling in your pipeline. Ensure accurate translation for low-resource languages.

Key insights

The USP system combines VLMs and NMT for culturally-aware image captioning in indigenous languages, revealing NMT vocabulary gaps.

Principles

• Cascade VLMs with NMT for complex tasks.
• Incorporate cultural prompts for relevance.
• Verify NMT model vocabulary coverage.

Method

Qwen3-VL-8B-Instruct generates Spanish captions with cultural prompts. Fine-tuned NLLB-200-distilled-600M models then translate to indigenous languages, trained on AmericasNLP 2023 and parallel corpora.

In practice

• Use Qwen3-VL-8B-Instruct for initial captioning.
• Fine-tune NLLB-200-distilled-600M for specific languages.
• Augment training data with parallel corpora.

Topics

• Image Captioning
• Indigenous Languages
• Vision-Language Models
• Neural Machine Translation
• Qwen3-VL-8B-Instruct
• NLLB-200-distilled-600M
• Low-Resource NLP

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Data Synthesis and Parameter-Efficient Fine-Tuning for Low-Resource NMT: A Case Study on Q'eqchi' Mayan — Synthetic data effectively bootstraps NMT structural learning for low-resource languages, but requires authentic data for semantic depth.
• Analysis of Machine Translators on Sentences Generated by Portuguese Image Captioning Models — Machine translation quality directly impacts image captioning model performance when adapting to new languages.
• Disparities In Negation Understanding Across Languages In Vision-Language Models — VLMs exhibit affirmation bias, with negation understanding varying significantly across typologically diverse languages.
• Multilingual Reasoning Cascades Need More Context — Translation cascades benefit significantly from preserving original language context throughout the pipeline.
• Bridging Cultural Gaps in Automated Translation of Brazilian Expressions: A Study on Cultural Adaptation — Automated translation systems struggle with cultural adaptation, even with advanced LLMs, requiring human oversight.
• WeGenBench: A Multidimensional Diagnostic Benchmark towards Text-to-Image Model Optimization — WeGenBench offers a multi-dimensional, VLM-integrated benchmark to precisely diagnose text-to-image model deficiencies across diverse linguistic and cultural contexts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.