A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

2026-07-23 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, quick

Summary

The CUP dataset, a new Greek book retrieval benchmark, comprises 868 catalog records and 104 expert-annotated queries with graded relevance judgments. This benchmark evaluates sparse (BM25), dense (sentence-transformers), hybrid, and LLM-assisted retrieval methods. Findings indicate that multilingual embeddings consistently outperform Greek-specific models, with hybrid retrieval achieving the best overall performance. A query-level analysis reveals BM25's strength in named-entity queries, while dense and hybrid methods significantly improve natural-language, noisy, cross-lingual, and concept queries. LLM TOC summarization enhances TOC-only retrieval, and LLM post-filtering improves early-stage retrieval, albeit at a high cost.

Key takeaway

For NLP engineers developing information retrieval systems for non-English languages like Greek, you should prioritize hybrid retrieval methods, especially those incorporating multilingual embeddings. This approach demonstrably outperforms language-specific models and pure sparse or dense methods across various query types. Evaluate your system's performance on diverse query categories, leveraging BM25 for named entities and hybrid methods for semantic or noisy queries, to optimize user experience and resource allocation.

Key insights

Hybrid retrieval, leveraging multilingual embeddings, offers superior performance for diverse Greek book search queries.

Principles

• Multilingual embeddings often surpass language-specific models for retrieval.
• Hybrid retrieval combines sparse and dense methods effectively.
• Query type dictates optimal retrieval method performance.

Method

The study evaluates sparse, dense, hybrid, and LLM-assisted retrieval methods using a new expert-annotated Greek book retrieval benchmark (CUP dataset) with graded relevance judgments.

In practice

• Prioritize hybrid retrieval for robust Greek book search systems.
• Use BM25 for named-entity queries.
• Consider LLM TOC summarization for TOC-only search.

Topics

• Information Retrieval
• Greek Language Processing
• Text Embeddings
• Hybrid Search
• LLM Applications
• BM25

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset — Hybrid retrieval combining multilingual embeddings and lexical methods offers superior performance for low-resource language book search.
• Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval — Embeddings provide synonym-tolerant search but predictably fail on structural issues like OOV terms and logical operations.
• Howard University-AI4PC at SemEval-2026 Task 8: Query Reformulation and Dense-Lexical Retrieval Fusion for Multi-Turn Retrieval-Augmented Generation — A training-free hybrid RAG system combines multi-signal query reformulation and fused retrieval for competitive multi-turn performance.
• The Case for Semantic Tokens in Modern Ranking Systems, How Embedding Size Affects Dense Retrieval, and More! — Semantic tokens and value vectors enhance large ranking systems and LLM embeddings, improving information retrieval.
• An End-to-End Ukrainian RAG for Local Deployment. Optimized Hybrid Search and Lightweight Generation — High-quality, verifiable Ukrainian RAG is achievable locally on resource-constrained hardware through optimized search and lightweight generation.
• Ranking Millions of Candidates with LLMs via Aggregated Embeddings, Converting Single-Vector Retrievers into Multi-Vector Models Without Retraining, and More! — Information retrieval research is rapidly advancing, focusing on LLM integration, multi-vector models, and efficiency at scale.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.