bLLeQA: Benchmarking LLMs for Grounded Legal Question-Answering in French and Dutch

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

bLLeQA is a new bilingual parallel question-answering dataset designed to benchmark large language models (LLMs) for grounded legal question-answering in French and Dutch. Developed to address resource scarcity for retrieval-augmented generation (RAG) systems in languages like Dutch, bLLeQA utilizes Belgian legal resources, providing aligned questions, answers, and supporting articles in both languages. This dataset facilitates the evaluation of both retrieval and end-to-end RAG pipelines. Benchmarking the full RAG pipeline in a zero-shot setting revealed that open-weight models match proprietary models in retrieval and citation extraction but underperform in generation quality. Critically, all models exhibit weak refusal capabilities, failing to detect incomplete supporting sources, and the end-to-end RAG setup still produces 20% flawed responses even in optimal conditions.

Key takeaway

For NLP Engineers developing legal RAG systems for multilingual European contexts, you should prioritize robust source verification and refusal mechanisms. While open-weight models are viable for retrieval, invest in refining their generation quality to match proprietary solutions. Be aware that even optimized RAG pipelines can yield 20% flawed responses, necessitating careful human oversight and continuous model improvement.

Key insights

RAG systems for legal Q&A in French and Dutch face significant challenges, especially in generation quality and source refusal.

Principles

• Open-weight LLMs compete in RAG retrieval.
• Proprietary LLMs lead in RAG generation.
• LLM refusal capabilities are generally weak.

Method

The study benchmarks full RAG pipelines in a zero-shot setting, evaluating retrieval, citation extraction, refusal behavior, and generation quality using the bLLeQA dataset.

In practice

• Use bLLeQA for legal RAG benchmarking.
• Prioritize improving LLM refusal behavior.
• Focus on RAG generation quality for open-weight models.

Topics

• Retrieval-Augmented Generation
• Legal Question Answering
• bLLeQA Dataset
• Multilingual NLP
• LLM Benchmarking
• Refusal Capability

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, NLP Engineer, Legal Professional

Related on AIssential

• Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents — Legal-DC and LegRAG advance Chinese legal RAG by providing specialized benchmarks and a framework for structured legal document processing.
• Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark — Monolingual LLMs for under-resourced languages require specific evaluation against multilingual counterparts.
• Large Language Models to Enhance Business Process Modeling: Past, Present, and Future Trends — LLMs significantly advance text-to-BPMN generation but face challenges in semantic accuracy, evaluation, and real-world applicability.
• Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions — Multi-Legal-Bench reveals LLM legal reasoning varies significantly across jurisdictions and tasks, with transfer quality tied to label-set alignment.
• When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering — LLMs require retrieval augmentation and strong reasoning for reliable off-guideline medical question answering.
• X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation — Multilingual RAG systems face significant challenges with contradictory evidence, requiring specialized diagnostic tools and pipelines like X-MADAM-RAG.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.