SimIdioms: A Corpus and Benchmark for Ukrainian Idiom Translation

2026-06-28 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

SimIdioms is a newly released corpus and benchmark designed for Ukrainian idiom translation, featuring aligned Ukrainian–English idiomatic expressions. The corpus was built by linking entries from multiple phraseological dictionaries and the MIDAS corpus via vector similarity search, then enhanced with figurative meanings, contextual sentences from the UberText fiction corpus, and semantic transparency scores. Researchers evaluated six large language models—Gemini 2.5 Flash, Claude Haiku 4.5, Gemma 3 12B, Qwen3-30B-A3B, LapaLM, and Tiny Aya Global—on 65,723 translations. The evaluation covered both Ukrainian-to-English and English-to-Ukrainian directions, using default and context-augmented prompting. Key findings include a significant directional asymmetry, with all models performing substantially worse when translating into Ukrainian. While providing figurative meaning and target idiom candidates improved Ukrainian-to-English translation quality for most models, its effect was limited in the reverse direction. Additionally, semantic transparency of idioms showed only a weak correlation with translation quality. The corpus and evaluation framework are publicly available to support research on idiomatic translation for mid-resource languages.

Key takeaway

For NLP Engineers and Research Scientists developing machine translation systems for mid-resource languages like Ukrainian, you should recognize the pronounced asymmetry in idiom translation performance. Current LLMs perform substantially worse when translating idioms into Ukrainian, and simply providing figurative meaning or target idiom candidates offers limited improvement in this direction. Prioritize research and development efforts on English-to-Ukrainian idiom translation, exploring more advanced contextualization techniques or specialized models to overcome this persistent challenge.

Key insights

LLMs exhibit significant directional asymmetry in idiom translation, performing worse into Ukrainian even with context.

Principles

• Idiom translation quality is highly direction-dependent.
• Contextual prompting benefits are direction-specific.
• Idiom semantic transparency weakly correlates with translation.

Method

Construct corpus by linking phraseological dictionaries via vector similarity, then enrich with figurative meanings, contextual sentences, and semantic transparency scores.

In practice

• Utilize SimIdioms for mid-resource language idiom research.
• Prioritize E-U idiom translation challenges.
• Consider figurative meaning for U-E translation prompts.

Topics

• Ukrainian Language
• Idiom Translation
• Large Language Models
• Machine Translation Benchmarking
• Corpus Development
• Mid-resource Languages

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Improving Machine Translation of Idioms: A Spanish–Galician Parallel Dataset and Synthetic Augmentation Approach — Synthetic data augmentation using LLMs significantly improves idiom translation in neural machine translation.
• Ukrainian Visual Word Sense Disambiguation Benchmark — Multimodal LLMs significantly underperform in Ukrainian Visual-WSD compared to English, revealing a critical language resource gap.
• G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment — Cross-lingual idiom alignment challenges LLMs due to non-compositionality, with glosses offering semantic grounding but literal bias remaining dominant.
• Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages — Idiomatic expressions challenge multilingual NLP, with literal meanings and low-resource languages posing significant hurdles.
• mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages? — LLMs demonstrate effective multilingual reasoning across 43 languages, with machine translations proving as effective as human ones.
• A Dataset for Probing Translationese Preferences in English-to-Swedish Translation — Language models often prefer literal "translationese" over idiomatic phrasing, especially when exposed to the source language.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.