Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

2025-12-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

A systematic evaluation characterizes effective language models (LMs) for multilingual supervised finetuning (SFT) data generation, addressing the common issue of suboptimal teacher selection. Researchers introduced the Polyglot Score, a metric combining intrinsic data quality (prompt diversity, length, response fluency, perplexity, reward score) and extrinsic student model performance across cultural understanding, mathematical reasoning, and general chat tasks. The study assessed 10 LMs across 6 diverse languages, generating over 1.4 million SFT examples and training 240 student models. Key findings indicate that Gemma 3 27B and Aya Expanse 32B are consistently effective teachers, often surpassing larger models like Llama 3.1 70B. Model scale alone does not predict teacher effectiveness; instead, data qualities such as prompt diversity, length, and response fluency explain over 93.3% of intrinsic data quality variance and predict student performance with R^2=0.664. Practical recommendations include matching teacher-student model families and strategically choosing data generation methods like Translate or Respond for less-resourced languages.

Key takeaway

For Machine Learning Engineers building multilingual language models, selecting an effective teacher for synthetic data generation is crucial. You should prioritize models like Gemma 3 27B or Aya Expanse 32B, as model scale alone does not guarantee superior data quality. Aligning your teacher and student model families can yield significant performance gains. For less-resourced languages, focus on "Respond" or "Translate" data generation methods to maximize synthetic data utility.

Key insights

Effective multilingual LM teachers are defined by synthetic data quality, not just model scale, and can be systematically evaluated.

Principles

• Model scale does not significantly predict multilingual teacher effectiveness.
• Data quality metrics (diversity, fluency, length) predict student performance.
• Teacher-student model family alignment improves knowledge transfer.

Method

The Polyglot Score aggregates intrinsic data quality (diversity, perplexity, reward score) and extrinsic student model performance (PGR on cultural, math, chat tasks) to holistically assess multilingual teacher LMs.

In practice

• Prioritize Gemma 3 27B or Aya Expanse 32B as multilingual teachers.
• Match teacher and student model families for improved performance.
• Use "Respond" or "Translate" methods for less-resourced languages.

Topics

• Multilingual LMs
• Synthetic Data Generation
• Supervised Finetuning
• Polyglot Score
• Teacher-Student Learning
• Gemma 3
• Aya Expanse

Code references

• ljvmiranda921/polyglot-teachers

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• AFRILANGTUTOR: Advancing Language Tutoring and Culture Education in Low-Resource Languages with Large Language Models — Dictionary-based synthetic data generation and combined SFT+DPO significantly enhance LLM tutoring for low-resource African languages.
• "Sorry, I Didn't Catch That": How Speech Models Miss What Matters Most — Speech models fail on high-stakes, short utterances despite low benchmark word error rates, especially for non-English speakers.
• Analysis of Machine Translators on Sentences Generated by Portuguese Image Captioning Models — Machine translation quality directly impacts image captioning model performance when adapting to new languages.
• Efficient Financial Language Understanding via Distillation with Synthetic Data — Distillation with synthetic data, guided by clustering real examples, enables efficient financial NLP in low-resource settings.
• I spent 4 days training an AI model to understand my country. It barely works — Low-resource NLP projects face significant challenges in data collection, translation, and accurate labeling, often yielding modest baselines.
• Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages — Multi-LCB extends code generation benchmarks beyond Python to expose LLM multilingual performance disparities and overfitting.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.