Professional Translators Versus Quality Estimation Models: Reliability and Agreement in English-Ukrainian Translation Evaluation

2026-06-28 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, short

Summary

A study evaluated English-Ukrainian translation quality by comparing professional human judgments with automatic Quality Estimation (QE) models and large language models (LLMs). Eight professional translators rated 1,000 sentence pairs on a 0-100 scale, using either a holistic scoring method or a two-stage fluency-plus-adequacy protocol. Professionals employing the holistic scale achieved significantly higher inter-rater reliability compared to those using separate fluency and adequacy scales, contradicting expectations that multidimensional evaluation improves agreement. Adequacy showed a strong correlation with holistic judgments, while fluency emerged as a largely independent dimension. The study also observed a leniency drift and increased evaluation speed among experts. Furthermore, three LLMs (Gemini 3 Flash, GPT-5.4, Gemma 3 27B) were assessed as quality judges. The larger models, GPT-5.4 and Gemma 3 27B, modestly outperformed dedicated QE models, achieving correlation coefficients of r = 0.814–0.821 against expert scores, compared to r ≤ 0.747 for QE models. LLMs also replicated the human pattern of adequacy dominating holistic quality perception.

Key takeaway

For NLP Engineers evaluating English-Ukrainian translation quality, you should prioritize holistic human evaluation over multi-dimensional scoring to achieve higher inter-rater reliability. When automating quality estimation, consider deploying advanced LLMs like GPT-5.4 or Gemma 3 27B, as they modestly outperform dedicated QE models in correlating with expert judgments. Focus your evaluation metrics on adequacy, as meaning preservation is the dominant factor in perceived translation quality for both human and machine judges.

Key insights

Holistic translation quality evaluation yields higher human agreement, driven by adequacy, a pattern replicated by advanced LLMs.

Principles

• Holistic quality scoring improves inter-rater reliability over multidimensional evaluation.
• Meaning preservation (adequacy) is the primary driver of perceived translation quality.
• Fluency is a largely independent dimension from overall translation quality.

In practice

• Implement holistic scoring for human translation quality assessment to enhance consistency.
• Deploy advanced LLMs (e.g., GPT-5.4, Gemma 3 27B) for automated translation quality estimation.
• Focus on adequacy metrics, as they strongly drive overall translation quality perception.

Topics

• Translation Quality Estimation
• Large Language Models
• English-Ukrainian Translation
• Inter-rater Reliability
• Human Evaluation
• Fluency and Adequacy

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Evaluating Small Language Models for English-to-Portuguese Translation: Impact of Model Scale and Quantization — 8-bit quantization maintains translation quality in SLMs, while model scale weakly correlates with performance.
• Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text — LLMs systematically distort diagnostic uncertainty in clinical text, often converting nuanced statements into definite assertions.
• Evaluative Judgement in Teaching AI-based Translation: A Class-room Case Study of AI-Mediated Translation and Post-Editing — Students prioritize human-centric factors like adequacy and post-editing effort over automatic metrics when evaluating AI-mediated translations.
• Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation — LLMs' verbalized confidence in translation outputs can be as reliable as internal signals, despite no correlation.
• PaliBench: A Multi-Reference Blueprint for Classical Language Translation Benchmarks — Multi-reference benchmarks are essential for evaluating classical language translation due to inherent interpretive plurality.
• Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field — Consistent evaluation reveals many reported SLT performance gains are artifacts of inconsistent implementation and evaluation setups.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.