Nearest-Neighbor Retrieval for Indigenous Image Captioning

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

NAIST's submission to the AmericasNLP 2026 Shared Task on Indigenous Language Image Captioning investigated two distinct approaches for generating captions in Bribri, Guaraní, Nahuatl, Wixárika, and Yucatec Maya. The first method employed a nearest-neighbor retrieval system, utilizing CLIP image embeddings to find the most similar image from a development set and directly reusing its caption. The second was a generation pipeline integrating scene analysis, dictionary-grounded lexical planning, retrieved gloss templates, and interlinear gloss representations for low-resource settings. The retrieval-based approach significantly outperformed the gloss-based pipeline in chrF++ evaluation, securing first-place automated system rankings for Bribri and Wixárika, and third place for Nahuatl. The gloss-based pipeline yielded weaker results, highlighting issues with dictionary coverage and grammatical instability.

Key takeaway

For NLP Engineers developing image captioning systems for low-resource indigenous languages, prioritize nearest-neighbor retrieval methods. Your initial efforts should focus on curating high-quality image-caption pairs, as this approach, leveraging CLIP embeddings, significantly outperforms complex generation pipelines. This strategy provides a robust baseline, potentially achieving top performance for languages like Bribri and Wixárika, while mitigating issues seen with dictionary coverage and grammatical stability in generative models.

Key insights

Nearest-neighbor retrieval using CLIP image embeddings offers a strong baseline for low-resource indigenous language image captioning.

Principles

• Retrieval methods excel in low-resource captioning.
• High-quality examples are key for retrieval success.
• Gloss-based pipelines struggle with resource gaps.

Method

The nearest-neighbor retrieval system uses CLIP image embeddings to identify the most similar image from a development set, directly reusing its associated caption for generation.

In practice

• Apply CLIP embeddings for image similarity.
• Curate high-quality example datasets.
• Benchmark retrieval for low-resource tasks.

Topics

• Nearest-Neighbor Retrieval
• Image Captioning
• Indigenous Languages
• CLIP Embeddings
• Low-Resource NLP
• AmericasNLP Shared Task

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Zero-Shot Captioning for Cultural Heritage: Automated Image Analysis of Traditional Indonesian Clothing — Retrieval-augmented vision-language models effectively generate culturally accurate captions for low-resource heritage data.
• Evaluation of Chunking Strategies for Effective Text Embedding in Low-Resource Language on Agricultural Documents — Effective text chunking significantly improves RAG performance in low-resource, morphologically complex languages.
• Connecting Speech to Words through Images — Spoken and written word mappings can be learned without text supervision by grounding them in shared visual contexts from images and their descriptions.
• Prodigy-ANN for Image Retrieval via CLIP — Multimodal CLIP embeddings enable efficient image retrieval and annotation by querying image databases with text.
• pplx-embed + Qdrant: Building Production-Grade Semantic Search with Quantization — Perplexity introduced `pplx-embed` on February 26, 2026, a new family of state-of-the-art multilingual text embedding models that utilize diffusion-based pretraining and are built on Qwen3.
• TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment — TEVI enhances vision-language alignment by using captions to selectively edit image embeddings via sparse autoencoders.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.