Evaluating Frontier LLM Translation Capability for Lakota

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Seven large language models, including four proprietary and three open-weight, were evaluated for bidirectional Lakota–English translation using 200 sentence pairs from the New Lakota Dictionary. Each model was tested with and without extended reasoning. Gemini 3.1 Pro performed best, achieving a mean chrF++ of 59.4 for Lakota→English and 42.6 for English→Lakota. Open-weight models lagged behind proprietary leaders, and no model delivered reliable translation. Independent LLM judges found semantic equivalence ranging from 6% (GPT-5.2) to 60% (Gemini), showing a substantial divergence from chrF++ scores. For open-weight models, reasoning primarily changed refusal behavior rather than improving translation quality. Diacritic analysis revealed models produce correct base characters but inconsistent diacritical marks. All results and evaluation code are publicly available.

Key takeaway

For NLP Engineers developing translation systems for low-resource languages like Lakota, you should recognize that even frontier LLMs like Gemini 3.1 Pro do not provide reliable translation. Relying solely on automated metrics like chrF++ is insufficient; incorporate independent semantic evaluation, potentially using LLM judges, to accurately assess quality and identify diacritic inconsistencies. Your development efforts should prioritize robust handling of diacritics and comprehensive human-like evaluation.

Key insights

Frontier LLMs struggle with reliable bidirectional Lakota-English translation, even with advanced reasoning.

Principles

• Proprietary LLMs generally outperform open-weight models in low-resource language translation.
• Extended reasoning primarily alters refusal behavior, not translation quality, for open-weight models.
• Automated metrics like chrF++ can diverge significantly from human-judged semantic equivalence.

Method

Evaluated seven LLMs on 200 Lakota-English sentence pairs using chrF++ and independent LLM judges (Cohen's κ=0.75) for semantic equivalence.

In practice

• LLM judges offer a valuable, distinct metric for translation quality assessment.
• Diacritic consistency remains a significant challenge for LLMs in low-resource languages.

Topics

• Large Language Models
• Machine Translation
• Lakota Language
• Low-Resource Languages
• LLM Evaluation
• Diacritics

Code references

• robotson/lakota-translation-benchmark

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Student

Related on AIssential

• Evaluating LLM-Based Translation of a Low-Resource Technical Language: The Medical and Philosophical Greek of Galen — LLMs translate Ancient Greek technical texts with high accuracy, but rare terminology significantly predicts failure.
• TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages — African language prompting reduces LLM refusal, but reveals comprehension and judge reliability issues in low-resource contexts.
• How Inference Compute Shapes Frontier LLM Evaluation — Frontier LLM evaluation performance is highly sensitive to inference compute and protocol choices.
• Can LLMs Write Correct TLA+ Specifications? Evaluating Natural-Language-to-TLA+ Generation — LLMs achieve only 8.6% semantic correctness for TLA+ specifications, primarily with progressive prompting.
• Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language — A compact 135M parameter LLM for Bangla achieves 1B-tier performance by efficiently merging tokens, overcoming low-resource script challenges.
• Linguistic Issues in Computational Linguistics for Non-English Languages — English-centric LLM design creates systematic linguistic weaknesses for non-English languages across multiple levels.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.