A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering

2026-03-12 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

This study presents a Retrieval-Augmented Generation (RAG) system for Khmer-language question answering within the telecom domain, addressing the challenge of low-resource, non-Latin-script languages. Researchers benchmarked three embedding models for dense retrieval, finding BGE-M3 (567M) superior with a Hit Rate@3 of 0.285 and File Hit Rate@3 of 0.700, significantly outperforming Jina-Embeddings-v3 (570M) and Qwen3-Embedding (597M). Subsequently, five locally deployable generator models—Qwen3 (8B), Qwen3.5 (9B), Sailor2-8B-Chat, SeaLLMs-v3-7B-Chat, and Llama-SEA-LION-v2-8B-IT—were evaluated on a 200-pair Khmer dataset using six RAGAS-inspired metrics. Results showed Qwen3.5-9B achieved highest faithfulness (0.859), Qwen3-8B led in factual correctness (0.380), and SeaLLMs-v3-7B-Chat excelled in answer relevance (0.867), answer similarity (0.836), and answer correctness (0.599). The study highlights that retriever performance remains a primary bottleneck, and generator strengths vary based on evaluation priorities like grounding or factual precision.

Key takeaway

For Machine Learning Engineers building RAG systems for low-resource languages like Khmer, you must prioritize robust retriever selection, as it remains a significant bottleneck. Do not assume English-centric RAG evaluation practices transfer directly; instead, focus on metrics like faithfulness and factual correctness, which better reflect expert judgment. Tailor your generator model choice (e.g., Qwen3.5-9B for grounding, Qwen3-8B for factual precision) to your application's specific quality requirements.

Key insights

RAG efficacy for low-resource Khmer hinges on robust retriever selection and generator alignment with specific quality objectives.

Principles

• Retriever performance is a primary bottleneck for low-resource RAG systems.
• Faithfulness and factual correctness are reliable RAG evaluation signals.
• Raw cosine similarity is an unreliable indicator of retrieval accuracy.

Method

Evaluate RAG in two phases: first, benchmark embedding models for dense retrieval; then, assess generator LLMs using RAGAS-inspired metrics on a domain-specific dataset.

In practice

• Consider BGE-M3 for Khmer dense retrieval due to its superior performance.
• Align generator choice with specific RAG goals, e.g., Qwen3.5-9B for grounding.
• Emphasize faithfulness and factual correctness in Khmer RAG evaluation.

Topics

• Retrieval-Augmented Generation
• Khmer NLP
• Low-Resource Languages
• RAG Evaluation Metrics
• Dense Retrieval
• Multilingual LLMs

Code references

• vibrantlabsai/ragas

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering — RAG efficacy for low-resource languages like Khmer is bottlenecked by retriever performance, with generator strengths varying across RAGAS metrics.
• Your AI Is Not Always Right. This Is What Makes It Reliable — AI reliability stems from real-time information retrieval, not just model scale or learned patterns.
• X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation — Multilingual RAG systems face significant challenges with contradictory evidence, requiring specialized diagnostic tools and pipelines like X-MADAM-RAG.
• Retrieval-Augmented Generation as a Bridge Between Classical NLP Pipelines and Modern LLM Systems — Retrieval-augmented generation (RAG) addresses the limitation of static knowledge in large language models (LLMs) by connecting them to external, up-to-date knowledge sources.
• Retrieval-Augmented Generation Must Move Beyond Factual Grounding to Represent Diverse Opinions — Current RAG systems exhibit factual bias, necessitating Opinion-Aware RAG to represent diverse perspectives by preserving aleatoric uncertainty.
• FIDES: Faithful Inference via Deep Evidence Signals for Retrieval-Memory Conflict in RAG — FIDES resolves RAG retrieval-memory conflict by selectively applying contrastive decoding based on token-level conflict concentration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.