Fluency and Faithfulness in Human and Machine Literary Translation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, short

Summary

A study analyzing 130,486 translated paragraphs from 106 novels across 16 languages investigates the balance between fluency and faithfulness in literary translation. Researchers measured fluency as "original-likeness" using a translationese classifier based on part-of-speech n-grams, and faithfulness with the COMET-KIWI metric. The analysis included human, Google Translate, and TranslateGemma outputs, controlling for paragraph length. A consistent negative correlation was found between fluency and faithfulness for both human and Google Translate translations. This tradeoff was weaker and often non-significant for TranslateGemma. These results highlight that segment length influences automatic evaluation and confirm an inherent tension between producing fluent literary text and preserving its original semantic content.

Key takeaway

For NLP Engineers developing or evaluating literary translation systems, recognize the inherent negative correlation between translation fluency and faithfulness. When optimizing models like TranslateGemma, prioritize explicit metrics for both aspects, as segment length significantly impacts automatic evaluation. You should consider whether your application demands higher fluency for readability or greater faithfulness for semantic preservation, and tune your evaluation strategies accordingly to avoid unintended compromises.

Key insights

Literary translation often presents a tradeoff where increased fluency correlates negatively with faithfulness.

Principles

• Fluency and faithfulness negatively correlate.
• Segment length impacts evaluation.
• LLM performance varies in this tradeoff.

Method

Fluency was measured via a translationese classifier on POS n-grams, while faithfulness used COMET-KIWI, controlling for paragraph length.

In practice

• Evaluate literary MT with both metrics.
• Consider segment length in MT evaluation.
• Compare LLM outputs to human baselines.

Topics

• Literary Translation
• Machine Translation
• Translation Evaluation
• Fluency-Faithfulness Tradeoff
• Large Language Models
• COMET-KIWI

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language — LLMs struggle with combined negation and figurative language, with prompt style significantly impacting performance.
• Better Literary Translation: A Multi-Aspect Data Generation and LLM Training Approach — Multi-aspect iterative refinement generates superior literary translation data for LLM training.
• Authorship Attribution in Multilingual Machine-Generated Texts — Multilingual authorship attribution for LLM-generated text is complex, with current monolingual methods showing limited cross-lingual transferability across diverse language families.
• PaliBench: A Multi-Reference Blueprint for Classical Language Translation Benchmarks — Multi-reference benchmarks are essential for evaluating classical language translation due to inherent interpretive plurality.
• G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment — Cross-lingual idiom alignment in LLMs benefits from explicit semantic glosses, yet literal translation bias remains a significant challenge.
• Evaluating Small Language Models for English-to-Portuguese Translation: Impact of Model Scale and Quantization — 8-bit quantization maintains translation quality in SLMs, while model scale weakly correlates with performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.