HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

2026-07-22 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

HalluTruthQA is a new fine-grained benchmark designed to evaluate hallucination in Arabic question answering, addressing limitations of existing response-level benchmarks. It comprises 2,400 expert-curated examples across four knowledge-intensive domains: Islamic knowledge, history, science, and geography. Each example includes an Arabic question, a model-generated answer, a verified reference, a binary hallucination label, six candidate answers for factual verification, character-level erroneous spans, human-written explanations, and macro/micro hallucination types. The benchmark was used to evaluate four open-source LLMs—Allam, Falcon-H1, Qwen32, and Silma—in a zero-shot setting across detection, span-level localization, factual verification, and explanation evaluation. Results indicate varied model performance across tasks, with best scores of 0.880 Macro-F1 for detection, 0.516 F1-Sp for localization, 0.852 LO-Score for factual verification, and 0.644 for explanation evaluation.

Key takeaway

For Machine Learning Engineers and AI Scientists developing or deploying Arabic LLMs, you should move beyond basic hallucination detection. Integrating fine-grained evaluation methods like those in HalluTruthQA will reveal specific model weaknesses in localizing, verifying, and explaining factual errors. This approach enables more targeted model improvements and builds greater trust in your Arabic QA systems, ensuring higher factual accuracy in critical applications.

Key insights

Hallucination evaluation must extend beyond detection to include localization, verification, and explanation of factual errors.

Principles

• Different hallucination tasks capture distinct LLM abilities.
• No single LLM achieves strongest performance across all hallucination evaluation tasks.

Method

HalluTruthQA pairs Arabic questions with model answers, verified references, binary hallucination labels, candidate answers, character-level erroneous spans, human explanations, and macro/micro hallucination types.

In practice

• Utilize HalluTruthQA for fine-grained Arabic QA hallucination assessment.
• Evaluate LLMs on detection, localization, verification, and explanation tasks.
• Access HalluTruthQA code, dataset, and scripts via GitLab.

Topics

• Arabic Question Answering
• Hallucination Detection
• LLM Evaluation
• Factual Verification
• HalluTruthQA
• Natural Language Processing

Code references

• nlpresearcher/HalluTruthQA

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering — Hallucination evaluation for Arabic LLMs demands fine-grained detection, localization, verification, and explanation, not just binary labels.
• The Geometry of Truth: Detecting LLM Hallucinations with Geometric Algebra — Type II hallucinations are context contradictions, dangerous due to their plausibility and standard similarity metric blindness.
• HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering — Comprehensive LLM hallucination evaluation requires fine-grained detection, localization, verification, and explanation, not just binary labels.
• A multilingual hallucination benchmark: MultiWikiQHalluA — Multilingual hallucination benchmarks reveal higher rates in smaller models and lower-resource languages.
• Zero-source LLM Hallucination Detection with Human-like Criteria Probing — HCPD detects LLM hallucinations in zero-source settings by emulating human multi-criteria reasoning for truthfulness, outperforming baselines.
• HalluScore: Large Language Model Hallucination Question Answering Benchmark — HalluScore is a new Arabic QA benchmark for evaluating LLM hallucination, emphasizing cultural and linguistic nuances.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.