IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions

2025-12-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

IdioLink is a new retrieval benchmark designed to evaluate how well language models understand idiomatic expressions beyond their surface form. Comprising 10,700 documents and 2,140 queries across 107 idioms, the benchmark tests models' ability to link idiomatic phrases to conceptually equivalent literal or paraphrased meanings. Initial evaluations of 24 embedding models, including BGE, E5, Contriever, and Qwen, reveal that current models struggle with this task, often relying on superficial lexical or topical similarities. The research demonstrates that augmenting queries with explicit instructions and employing span embedding significantly improves zero-shot retrieval performance, with combined gains of +21.99 R-Precision and +22.71 nDCG on average. Fine-tuning further enhances results, but surprisingly, instruction augmentation and span embedding during training proved detrimental, suggesting a need to decouple training and inference strategies.

Key takeaway

For Machine Learning Engineers developing retrieval systems, this research highlights that current embedding models struggle with idiomatic language, relying on surface-level cues. You should prioritize explicit instruction augmentation and span embedding at inference time to significantly improve conceptual retrieval performance, even with smaller models. Be cautious when fine-tuning; standard query and sentence embedding configurations are more effective during training, suggesting a need to separate training and inference strategies for optimal results.

Key insights

Idiom understanding in retrieval requires semantic abstraction beyond lexical overlap.

Principles

• Explicit query instructions enhance retrieval.
• Span-level embeddings improve fine-grained matching.
• Model scale does not guarantee performance.

Method

IdioLink formulates retrieval by matching queries (literal/idiomatic PIE usage) to documents (literal, idiomatic, simplification, sense) conveying the same core meaning, without document span access.

In practice

• Prepend instructions to queries for better zero-shot.
• Use span embedding for localized meaning.
• Decouple training/inference strategies.

Topics

• Idiomatic Expressions
• Semantic Retrieval
• Language Models
• Embedding Models
• Natural Language Processing
• Benchmark Datasets

Code references

• Intellexus-DSI/IdioLink
• FullStackRetrieval-com/RetrievalTutorials

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressions — Idioms challenge language models by requiring semantic abstraction beyond surface-level lexical overlap.
• Rethinking Negative Sampling for Knowledge Distillation in Retrieval, A Distillation Recipe for Small Language Model Search Agents, and More! — Effective information retrieval advancements often stem from rethinking fundamental assumptions and optimizing data representation or model behavior.
• The Birth of Meaning — Word embeddings convert text into dense numerical vectors, capturing semantic meaning and context for AI models.
• From TF-IDF to Transformers: Implementing Four Generations of Semantic Search — Semantic search evolved from explicit rules to learned contextual representations, balancing interpretability with semantic depth.
• DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark — DocRetriever improves multimodal document retrieval via layout-aware sparse embeddings and a generalizable, reasoning-augmented reranker.
• FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings — FLiP models effectively interpret sentence embeddings by linearly extracting lexical content, revealing underlying biases.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.