A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books

2025-11-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

A novel pipeline extracts grammatical rules, example sentences, and lexicons from descriptive grammar books using large language models to generate synthetic parallel corpora. This data is then used for fine-tuning machine translation models, a departure from prior work that used grammar content for inference-time prompting. Validated on Kalamang (Papuan), Tuatschin (Romance), and Mandan (Siouan), the approach improved over seed-data baselines in 75% of configurations for Kalamang and 59% for Tuatschin, yielding best-case ChrF++ gains of +8.8, +5.3, and +3.3 respectively. A systematic factorial study across 96 configurations, varying target part-of-speech, retrieval granularity, and sample volume, identified optimal factor combinations. The pipeline, which fine-tunes Gemini-2.5-flash, requires only ~200 annotated lines per language for its BERT-based sentence classifier.

Key takeaway

For Machine Learning Engineers building translation systems for severely under-resourced languages, this research offers a practical methodology. You should consider implementing a grammar book-driven synthetic data generation pipeline to fine-tune models like Gemini-2.5-flash. Carefully evaluate retrieval granularity, target part-of-speech, and sample volume. These factors significantly impact translation quality and can degrade performance if not optimized.

Key insights

Grammar books can be semi-automatically processed to generate synthetic parallel data for low-resource MT fine-tuning.

Principles

• Synthetic data improves MT for low-resource languages.
• Effectiveness varies by language, POS, and data volume.
• Excessive synthetic data can degrade model performance.

Method

A semi-automated pipeline extracts rules, lexicons, and parallel sentences from PDFs using LLMs and a BERT classifier. It then generates synthetic parallel corpora via rule-guided lexical substitution for fine-tuning.

In practice

• Use rule-guided lexical substitution for MT data augmentation.
• Experiment with retrieval granularity (rule vs. section).
• Monitor synthetic data volume to avoid performance degradation.

Topics

• Low-Resource MT
• Synthetic Data Generation
• Grammar Books
• Large Language Models
• Model Fine-tuning
• Endangered Languages

Code references

• varunghat/GrammarMT

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books — Grammar books can be systematically converted into synthetic parallel data for low-resource machine translation fine-tuning.
• "Sorry, I Didn't Catch That": How Speech Models Miss What Matters Most — Speech models fail on high-stakes, short utterances despite low benchmark word error rates, especially for non-English speakers.
• Ancient Greek to Modern Greek Machine Translation: A Novel Benchmark and Fine-Tuning Experiments on LLMs and NMT Models — A new AG-MG parallel corpus and benchmark significantly advance Ancient Greek to Modern Greek machine translation.
• I spent 4 days training an AI model to understand my country. It barely works — Low-resource NLP projects face significant challenges in data collection, translation, and accurate labeling, often yielding modest baselines.
• Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation — Effective multilingual LM teachers are defined by synthetic data quality, not just model scale, and can be systematically evaluated.
• Efficient Financial Language Understanding via Distillation with Synthetic Data — Distill LLM instruction-following to compact models using synthetic data generated from minimal, semantically clustered seeds.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.