Data Contamination in Neural Hieroglyphic Translation: A Reproducibility Study

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A reproducibility study on neural hieroglyphic-to-German translation revealed significant data contamination issues. While a prior study reported 61.5 BLEU using fine-tuned M2M-100, this reproduction achieved only 37.0 BLEU. The discrepancy stems from 32% of test targets (16/50) appearing identically in the training data, with 50% showing 8-gram overlap at a 70% threshold. This contamination inflated scores dramatically, yielding 83.8 BLEU / 0.924 COMET-22 on contaminated samples versus 30.9–39.2 BLEU / 0.622–0.676 COMET-22 on clean ones. Document-level decontamination was insufficient, reducing contaminated BLEU by only 4.6 points, as 8/16 targets persisted via other source documents. Target-level deduplication is essential. The study releases a decontaminated 34-sample test set and establishes corrected baselines of 30.9–39.2 BLEU, offering a realistic assessment of NMT for this endangered writing system.

Key takeaway

For NLP Engineers evaluating Neural Machine Translation models, especially for low-resource or ancient languages, you must rigorously audit your test sets for data contamination. Relying on inflated BLEU scores from contaminated data will lead to overestimating model capabilities and misallocating development resources. Prioritize target-level deduplication to ensure realistic performance assessments and avoid misleading benchmarks.

Key insights

Data contamination significantly inflates NMT scores for scarce languages, requiring rigorous deduplication.

Principles

• Scarce language NLP datasets are highly susceptible to contamination.
• Test set contamination dramatically inflates NMT performance metrics.
• Target-level deduplication is crucial for accurate evaluation.

Method

The study investigated a BLEU score gap by reproducing a neural hieroglyphic translation model, identifying test-training data overlap, and then decontaminating the test set to establish corrected baselines.

In practice

• Audit test sets for identical target-training overlaps.
• Implement target-level deduplication for NMT evaluation.
• Use the released 34-sample decontaminated test set.

Topics

• Neural Machine Translation
• Data Contamination
• Low-Resource Languages
• Hieroglyphic Translation
• Reproducibility Study
• BLEU Score

Best for: Research Scientist, AI Scientist, NLP Engineer, AI Student

Related on AIssential

• Twenty Years of HAREM: A Reproducible Audit and Reassessment of Portuguese Named Entity Recognition — Data contamination in benchmarks significantly distorts model performance and masks true generalization capabilities.
• Detecting Translation Hallucinations with Attention Misalignment — Bidirectional attention map comparison offers an interpretable and efficient way to detect NMT hallucinations.
• Data Synthesis and Parameter-Efficient Fine-Tuning for Low-Resource NMT: A Case Study on Q'eqchi' Mayan — Synthetic data effectively bootstraps NMT structural learning for low-resource languages, but requires authentic data for semantic depth.
• LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training — Data contamination in RL post-trained LLMs can be reliably detected by analyzing layer-wise representation deviations.
• Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field — Consistent evaluation reveals many reported SLT performance gains are artifacts of inconsistent implementation and evaluation setups.
• A Calibrated Memorization Index (MI) for Detecting Training Data Leakage in Generative MRI Models — A new metric effectively detects training data duplication in generative MRI models, enhancing privacy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.