A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering

2026-05-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

A comparative study evaluated a Retrieval-Augmented Generation (RAG) system for Khmer-language telecom documents, addressing its efficacy for low-resource, non-Latin-script languages, through a two-phase evaluation. First, three embedding models—BGE-M3 (567M), Jina-Embeddings-v3 (570M), and Qwen3-Embedding (597M)—were benchmarked for dense retrieval. BGE-M3 consistently outperformed others, achieving a Hit Rate@3 of 0.285, File Hit Rate@3 of 0.700, MRR@3 of 0.221, and Precision@3 of 0.112. Second, using BGE-M3 as the retriever, five generator backends—Qwen3 (8B), Qwen3.5 (9B), Sailor2-8B-Chat, SeaLLMs-v3-7B-Chat, and Llama-SEA-LION-v2-8B-IT—were assessed on 200 Khmer question-answer pairs using six RAGAS-inspired metrics. No single generator dominated; Qwen3.5-9B led in faithfulness (0.859) and context relevance (0.726), Qwen3-8B in factual correctness (0.380), while SeaLLMs-v3-7B-Chat performed best on answer relevance (0.867), similarity (0.836), and correctness (0.599). Retriever choice remains a major bottleneck for Khmer RAG; generator strengths vary by priority.

Key takeaway

For NLP Engineers building Retrieval-Augmented Generation systems for low-resource, non-Latin languages like Khmer, your retriever choice is a critical bottleneck. You should prioritize robust embedding models such as BGE-M3, which demonstrated superior dense retrieval. Furthermore, align your generator selection with specific RAGAS metrics: use Qwen3.5-9B for faithfulness, Qwen3-8B for factual correctness, or SeaLLMs-v3-7B-Chat for answer relevance and similarity, based on your application's primary objective.

Key insights

RAG efficacy for low-resource languages like Khmer is bottlenecked by retriever performance, with generator strengths varying across RAGAS metrics.

Principles

• Retriever selection is critical for low-resource RAG.
• Generator performance is metric-dependent.
• Optimize generators for specific RAGAS priorities.

Method

A two-phase comparative evaluation benchmarks embedding models for dense retrieval, then assesses generator backends using RAGAS-inspired metrics on a curated golden dataset.

In practice

• Consider BGE-M3 for Khmer dense retrieval.
• Use Qwen3.5-9B for high faithfulness.
• Prioritize SeaLLMs-v3-7B-Chat for answer relevance.

Topics

• Retrieval-Augmented Generation
• Khmer Language Processing
• Embedding Models
• Language Model Benchmarking
• RAGAS Metrics
• Dense Retrieval

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering — RAG efficacy for low-resource Khmer hinges on robust retriever selection and generator alignment with specific quality objectives.
• Retrieval-Augmented Generation as a Bridge Between Classical NLP Pipelines and Modern LLM Systems — Retrieval-augmented generation (RAG) addresses the limitation of static knowledge in large language models (LLMs) by connecting them to external, up-to-date knowledge sources.
• Top 5 Embedding Models for Your RAG Pipeline — Effective RAG pipelines rely on embedding models that balance retrieval performance, real-world adoption, and deployment practicality.
• Building a PDF Question-Answering Chatbot with Spring AI: From PDF Upload to RAG-Powered Answers — RAG systems enhance LLM accuracy by grounding answers in retrieved, relevant private data, outperforming larger models without context.
• Retrieval-Augmented Generation Must Move Beyond Factual Grounding to Represent Diverse Opinions — Current RAG systems exhibit factual bias, necessitating Opinion-Aware RAG to represent diverse perspectives by preserving aleatoric uncertainty.
• Evaluation of Chunking Strategies for Effective Text Embedding in Low-Resource Language on Agricultural Documents — Effective text chunking significantly improves RAG performance in low-resource, morphologically complex languages.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.