A Logic-Based Approach to Hallucinations in Data-to-Text NLG: Experiments with Human and LLM Annotators

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A logic-based framework, introduced by van Deemter (2024), categorizes hallucinations in data-to-text natural language generation (NLG) into seven disjoint types based on the relation of logical consequence. This study investigated the framework's applicability using both human annotators and large language models across two distinct data-to-text domains. The findings indicate that the framework is indeed applicable for identifying and classifying hallucinations. However, the research also revealed notable variations dependent on the specific domain, alongside significant discrepancies between the judgments made by human annotators and those from the large language models. These observations highlight critical issues that should guide future research efforts in hallucination detection and mitigation.

Key takeaway

For NLP Engineers developing data-to-text NLG systems, understanding hallucination types is crucial. You should consider adopting a structured, logic-based framework to categorize and analyze hallucinations, recognizing that its application may vary across domains. Be aware of potential discrepancies between human evaluations and LLM-based annotation tools, and plan for hybrid evaluation strategies to ensure robust hallucination detection and mitigation in your models.

Key insights

A logic-based framework categorizes data-to-text NLG hallucinations into seven types, applicable by humans and LLMs despite domain and judgment discrepancies.

Principles

• Hallucinations can be logically categorized.
• Framework applicability varies by domain.
• Human and LLM judgments diverge on hallucinations.

Method

Evaluated a 7-category logic-based hallucination framework using human and LLM annotators across two data-to-text domains to assess applicability and consistency.

Topics

• Natural Language Generation
• Data-to-Text
• LLM Hallucinations
• Annotation Frameworks
• Logical Consequence

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models — Linguistic tone systematically influences VLM hallucination, with varying effects on frequency and intensity across models and tasks.
• NTS-CoT: Mitigating Hallucinations in LLM-based News Timeline Summarization with Chain-of-Thought Reasoning — NTS-CoT uses Chain-of-Thought reasoning to reduce hallucinations and omissions in LLM-based news timeline summarization.
• Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection — Graph-based RAG hallucination detection is model-dependent, reversing for GPT-class models due to differing hallucination styles.
• Why AI Makes Things Up: Understanding Hallucinations in Language Models — LLM hallucinations are a structural outcome of probabilistic training, not a fixable bug, requiring systemic mitigation.
• Beyond Literal Summarization: Redefining Hallucination for Medical SOAP Note Evaluation — Lexical faithfulness in LLM evaluation misclassifies valid clinical reasoning as hallucination, inflating error rates.
• The Geometry of Truth: Detecting LLM Hallucinations with Geometric Algebra — Type II hallucinations are context contradictions, dangerous due to their plausibility and standard similarity metric blindness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.