HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

2026-07-22 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

HalluTruthQA is a new fine-grained benchmark designed for evaluating hallucination in Arabic question answering by large language models (LLMs). It addresses limitations of existing benchmarks by enabling detection, localization, explanation, and verification of factual errors. The benchmark comprises 2,400 expert-curated examples across Islamic knowledge, history, science, and geography. Each entry includes an Arabic question, a model-generated answer, a verified reference answer, a binary hallucination label, six candidate answers for factual verification, character-level erroneous spans, human-written explanations, and macro and micro hallucination types. Evaluations of four open-source LLMs—Allam, Falcon-H1, Qwen32, and Silma—in zero-shot settings showed varied performance, with best scores of 0.880 Macro-F1 for detection, 0.516 F1-Sp for localization, 0.852 LO-Score for factual verification, and 0.644 for explanation evaluation. This highlights the need to move beyond simple detection.

Key takeaway

For NLP engineers developing or evaluating Arabic LLMs for factual accuracy, relying solely on response-level hallucination detection is insufficient. You should adopt a fine-grained evaluation approach that includes error localization, factual verification, and explanation to truly understand and mitigate model shortcomings. This will lead to more robust and trustworthy Arabic language models.

Key insights

Comprehensive LLM hallucination evaluation requires fine-grained detection, localization, verification, and explanation, not just binary labels.

Principles

• LLM hallucination evaluation needs multi-faceted approaches.
• Different tasks capture distinct LLM abilities.
• Response-level labels are insufficient for analysis.

Method

The benchmark provides questions, model answers, verified references, binary labels, candidate answers, character-level error spans, human explanations, and macro/micro hallucination types.

In practice

• Implement character-level span detection for error localization.
• Incorporate human-written explanations for deeper error analysis.
• Evaluate models across detection, localization, verification, and explanation.

Topics

• HalluTruthQA
• Arabic Question Answering
• LLM Hallucination
• Factual Verification
• Error Localization
• Benchmark

Code references

• nlpresearcher/HalluTruthQA

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• HalluScore: Large Language Model Hallucination Question Answering Benchmark — HalluScore is a new Arabic QA benchmark for evaluating LLM hallucination, emphasizing cultural and linguistic nuances.
• HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering — Hallucination evaluation must extend beyond detection to include localization, verification, and explanation of factual errors.
• MultiHaluDet: Multilingual Hallucination Detection via LLM Hidden State Probing — MultiHaluDet detects multilingual LLM hallucinations by probing hidden states across layers without language-specific fine-tuning.
• Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AI — Islamic LLMs require deep domain grounding, precise citation, and nuanced disagreement representation for trustworthiness beyond mere Arabic fluency.
• HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models — Fine-grained hallucination diagnosis in MLLMs, combining detection, classification, and explanation, significantly improves correction over coarse methods.
• Language Model Hallucination Evaluation with GraphEval — GraphEval uses knowledge graphs and NLI to explainably detect and localize LLM hallucinations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.