From Machine Translation to Image Captioning: Training Vision-Language Models for Indigenous Languages of the Americas

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Computer Vision · Depth: Expert, quick

Summary

A system developed for the AmericasNLP 2026 Shared Task addresses cultural image captioning for Indigenous Languages of the Americas. This post-training pipeline begins with Aya Vision 32B, which is initially fine-tuned on machine translation data from previous AmericasNLP tasks. Subsequently, it undergoes further fine-tuning using cultural image captioning data. This two-stage approach uses translation as an intermediate training step, enabling the final system to generate captions directly in the target Indigenous language, rather than translating from Spanish. Experiments confirm that machine translation fine-tuning is a crucial initialization, and the resulting vision-language model also exhibits translation capabilities. Notably, a zero-shot GPT-5.5 submission secured first place in the Maya language track during official human evaluation.

Key takeaway

For NLP engineers developing vision-language models for low-resource Indigenous languages, consider a two-stage fine-tuning strategy. First, pre-train your base VLM on machine translation data, then fine-tune it on specific image captioning datasets. This method, demonstrated with Aya Vision 32B, effectively initializes the model and enables direct caption generation in the target language, potentially outperforming translation-based post-processing.

Key insights

Intermediate machine translation fine-tuning significantly improves vision-language models for Indigenous language image captioning.

Principles

• Translation fine-tuning aids VLM initialization.
• Direct Indigenous language captioning is feasible.

Method

Fine-tune Aya Vision 32B on machine translation data from prior AmericasNLP tasks, then further fine-tune on cultural image captioning data.

In practice

• Develop direct Indigenous language captioning.
• Explore VLM translation capabilities.

Topics

• Vision-Language Models
• Image Captioning
• Indigenous Languages
• Machine Translation
• Fine-tuning
• Aya Vision 32B
• AmericasNLP

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Analysis of Machine Translators on Sentences Generated by Portuguese Image Captioning Models — Machine translation quality directly impacts image captioning model performance when adapting to new languages.
• Finetuning Vision-Language-Action Models Requires Fewer Layers Than You Think — Vision-Language-Action models can be significantly compressed by removing redundant layers identified via a training-free method.
• Improving low-resource ASR using bilingual fine-tuning with language identification: a cross-linguistic evaluation — Bilingual fine-tuning enhances low-resource ASR, particularly when language identification is accurate or explicitly guided during inference.
• Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioning — Retrieval-augmented image captioning uses hierarchical multi-modal knowledge retrieval and LLMs to generate context-rich news image descriptions.
• GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models — Naively using VLM-generated captions can degrade accuracy due to anchoring effects and asymmetric error types.
• Multilingual Coreference Resolution via Cycle-Consistent Machine Translation — Coreference resolution in low-resource languages can be significantly improved by cycle-consistent machine translation data augmentation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.