A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, long

Summary

A new Greek book retrieval benchmark, CUP, has been introduced, comprising 868 catalog records and 104 expert-annotated queries with graded relevance judgments. This benchmark evaluates sparse (BM25), dense (sentence-transformers), hybrid, and LLM-assisted retrieval methods. Key findings indicate that strong multilingual embedding models consistently outperform Greek-specific ones, with the best Greek model (xlm-r) reaching only 0.372 nDCG@9 compared to several multilingual models exceeding 0.50. Weighted hybrid retrieval achieved the best overall performance (0.673 nDCG@9, 0.810 MRR), demonstrating complementarity between lexical and semantic cues. BM25 excels at named-entity queries, while dense and hybrid methods significantly improve natural-language, noisy, cross-lingual, and concept queries. LLM-based TOC summarization improved retrieval, and LLM post-filtering (qwen3) enhanced early precision (0.821 MRR) but incurred a high inference cost (avg 92.5 secs per query).

Key takeaway

For NLP engineers building retrieval systems for low-resource languages like Greek, you should prioritize weighted hybrid retrieval, as it significantly outperforms standalone sparse or dense methods, achieving 0.673 nDCG@9. Employ strong multilingual embedding models, which consistently surpass Greek-specific encoders. While LLM post-filtering can boost early precision, evaluate its high inference cost (avg 92.5 secs) against your real-time requirements. Consider LLM-based TOC summarization to enrich sparse metadata.

Key insights

Hybrid retrieval combining multilingual embeddings and lexical methods offers superior performance for low-resource language book search.

Principles

• Multilingual embeddings scale better than Greek-specific models.
• Hybrid retrieval effectively combines lexical and semantic signals.
• LLM-based TOC summarization can enrich document representations.

Method

The study developed a Greek book retrieval benchmark (CUP) with expert-annotated queries, evaluated sparse, dense, and hybrid methods, and explored LLM-based TOC summarization and post-filtering techniques.

In practice

• Prioritize multilingual embeddings over smaller, language-specific models.
• Implement weighted hybrid retrieval for robust performance.
• Consider LLM-generated TOC summaries to enhance document fields.

Topics

• Greek Information Retrieval
• Multilingual Embeddings
• Hybrid Retrieval
• LLM Applications
• Book Search
• CUP Dataset

Code references

• beir-cellar/beir

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset — Hybrid retrieval, leveraging multilingual embeddings, offers superior performance for diverse Greek book search queries.
• Ancient Greek to Modern Greek Machine Translation: A Novel Benchmark and Fine-Tuning Experiments on LLMs and NMT Models — A new AG-MG parallel corpus and benchmark significantly advance Ancient Greek to Modern Greek machine translation.
• Discovering Lexical Gaps Using Embeddings from Multilingual LLMs — Cross-lingual lexical gaps can be identified by measuring semantic alignment weakness in multilingual LLM embeddings.
• Granite Embedding Multilingual R2: Open Apache 2.0 Multilingual Embeddings with 32K Context — Best Sub-100M Retrieval Quality — New IBM Granite multilingual embedding models offer superior retrieval quality and extended context for diverse language and code applications.
• Anchor Detection for RAG: Parallel Detectors, Then One LLM Call at the End — Enterprise RAG retrieval combines parallel keyword and embedding detectors, aggregating results for a single, auditable LLM ranking.
• Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models — Cross-lingual fairness and tokenization efficiency are not fundamentally at odds in multilingual LLMs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.