Schema-Constrained Image Captioning for Five Low-Resource Indigenous Languages

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A novel LLM-assisted rule-based machine translation (LLM-RBMT) captioner was developed for schema-constrained image captioning across five low-resource indigenous languages: Bribri, Guaraní, Yucatec Maya, Orizaba Nahuatl, and Wixárika. This system, submitted to the AmericasNLP 2026 Shared Task, employs a coding agent to generate complete Pydantic grammar packages with closed vocabularies from small development splits and open-web linguistic references. During inference, a vision-language model (VLM) utilizes these schemas with constrained decoding to produce structured SentenceList outputs, which a deterministic Python renderer converts into surface strings. Notably, the model never directly generates target-language tokens. The architecture is consistent across all five languages, requiring no fine-tuning, parallel corpora, or human grammar edits. It secured first place in human evaluation for Bribri and Orizaba Nahuatl, third for Yucatec Maya, and first in ChrF++ for Yucatec Maya.

Key takeaway

For NLP Engineers working with low-resource languages or seeking robust grammar control in generation, this LLM-assisted rule-based machine translation (LLM-RBMT) approach offers a powerful alternative to traditional neural machine translation. You should consider adopting schema-constrained decoding with automatically generated grammars to achieve high-quality, reliable outputs, especially where data scarcity is a major challenge. This method ensures grammatical correctness by construction and eliminates the need for parallel corpora or extensive fine-tuning, minimizing the risk of ungrammatical or nonsensical generations.

Key insights

LLM-generated grammars and constrained decoding enable high-quality, grammatically correct image captions for low-resource languages without direct target-language token generation.

Principles

• Grammar-guided generation ensures correctness.
• Constrained decoding avoids direct target token generation.
• LLMs can automate grammar package creation.

Method

A coding agent generates Pydantic grammars from linguistic references. A VLM uses constrained decoding with these schemas to output structured SentenceList objects, which a Python renderer converts to surface strings.

In practice

• Apply to other low-resource NLP tasks.
• Use Pydantic for grammar definition.
• Explore LLM agents for grammar synthesis.

Topics

• Low-Resource Languages
• Image Captioning
• LLM-RBMT
• Constrained Decoding
• Pydantic Grammars
• AmericasNLP Shared Task

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Ancient Greek to Modern Greek Machine Translation: A Novel Benchmark and Fine-Tuning Experiments on LLMs and NMT Models — A new AG-MG parallel corpus and benchmark significantly advance Ancient Greek to Modern Greek machine translation.
• Beyond Surface Forms: A Comprehensive, Mechanism-Oriented Taxonomy of Indirect Linguistic Encoding for LLM-Based Coded Language Detection — A mechanism-oriented taxonomy of indirect linguistic encoding significantly enhances LLM-based coded language detection.
• Forget LLM: MIT's New RLM (Phase Shift in AI) — MIT's Recursive Language Model (RLM) uses a neurosymbolic approach to overcome LLM context rot and enhance reasoning.
• IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages — IndicDB reveals a significant "Indic Gap" in LLM Text-to-SQL performance for Indian languages compared to English.
• Linguistic Issues in Computational Linguistics for Non-English Languages — English-centric LLM design creates systematic linguistic weaknesses for non-English languages across multiple levels.
• TalkTag: Fine-Grained Morphosyntactic Error Annotation for Transcribed Speech — TalkTag automates fine-grained morphosyntactic error annotation in spoken transcripts using an LLM, even with scarce data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.