HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

HalluTruthQA introduces a fine-grained benchmark for evaluating hallucination in Arabic question answering, comprising 2,400 expert-curated examples across Islamic knowledge, history, science, and geography. Each entry features an Arabic question, a Fanar-generated answer, a verified reference, a binary hallucination label, character-level erroneous spans, human-written explanations, and six candidate answers for factual verification. The benchmark evaluates four open-source LLMs—Allam, Falcon-H1, Qwen32, and Silma—in a zero-shot setting across detection, span-level localization, factual verification, and explanation tasks. Results indicate no single model achieves top performance across all tasks, with Allam leading detection (0.880 Macro-F1) and explanation (0.644), Qwen32 excelling in localization (0.516 F1-Sp), and Falcon-H1 in factual verification (0.852 LO-Score). The study emphasizes moving beyond simple detection to granular error analysis.

Key takeaway

For NLP engineers developing or deploying Arabic LLMs for knowledge-intensive QA, you must move beyond simple binary hallucination detection. Your evaluation and mitigation strategies should integrate span-level localization, factual verification, and explanation quality metrics. Focus on addressing specific error patterns, such as temporal/numerical normalization and source attribution, which current models struggle with, to significantly enhance factual reliability.

Key insights

Hallucination evaluation for Arabic LLMs demands fine-grained detection, localization, verification, and explanation, not just binary labels.

Principles

• LLM hallucination tasks (detection, localization, verification, explanation) capture distinct abilities.
• Domain-specific knowledge areas introduce unique hallucination patterns and evaluation challenges.
• Multi-layer annotation schemes are crucial for granular hallucination analysis.

Method

A two-pass annotation process by domain experts and research assistants curates 2,400 QA examples with binary labels, character-level spans, explanations, and six candidate answers.

In practice

• Implement multi-dimensional benchmarks for comprehensive LLM factual reliability assessment.
• Prioritize span-level localization and explanation generation in Arabic QA system development.
• Tailor hallucination mitigation strategies to address domain-specific error types.

Topics

• Hallucination Detection
• Arabic LLMs
• Question Answering
• Benchmark Datasets
• Factual Verification
• Span Localization

Code references

• nlpresearcher/HalluTruthQA

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering — Hallucination evaluation must extend beyond detection to include localization, verification, and explanation of factual errors.
• The Geometry of Truth: Detecting LLM Hallucinations with Geometric Algebra — Type II hallucinations are context contradictions, dangerous due to their plausibility and standard similarity metric blindness.
• Evaluating Reference-Free Summarization Quality Metrics for Portuguese: A Study with Human Judgments in Financial News — Decomposing summarization evaluation into atomic QA pairs surpasses holistic LLM-as-a-Judge methods for Portuguese financial news.
• SAUCE: Summary Analysis Using Conversation Entailment — SAUCE provides an explainable, fact-based method for evaluating cross-lingual conversational speech summaries, identifying hallucinations and omissions.
• HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering — Comprehensive LLM hallucination evaluation requires fine-grained detection, localization, verification, and explanation, not just binary labels.
• A multilingual hallucination benchmark: MultiWikiQHalluA — Multilingual hallucination benchmarks reveal higher rates in smaller models and lower-resource languages.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.