Revisiting Faithfulness Annotations for Long-form Summaries

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, short

Summary

Yang Zhong, Yang Janet Liu, and Diane Litman's paper, presented at LAW XX in July 2026, examines faithfulness annotations for long-form summaries. This research addresses critical challenges in annotating faithfulness errors within discourse-rich summaries. These summaries serve as gold-standard benchmarks for evaluating language model faithfulness-checking systems. Using a discourse-aware evaluation framework and human auditing across three benchmarks, the authors found 3.4%-5.4% of sentence-level labels required revision. This was due to overlooked discourse-level inconsistencies. They introduce a taxonomy of five recurring annotation error types. Correcting these labels significantly alters system rankings, leading to recommendations for improved future annotation practices.

Key takeaway

For NLP Engineers developing or evaluating faithfulness-checking systems for long-form summaries, you should critically examine the underlying benchmarks. Your system's performance rankings may be skewed by unreliable annotations, as 3.4%-5.4% of labels can be inconsistent. Consider adopting discourse-aware annotation practices and auditing existing datasets to ensure the validity of your evaluations and comparisons. This will lead to more robust and accurate system development.

Key insights

Existing long-form summary faithfulness benchmarks contain significant discourse-level annotation errors impacting system evaluations.

Principles

• Discourse-level context is crucial for accurate faithfulness annotation.
• Standard annotation procedures often overlook complex inconsistencies.
• Benchmark reliability directly impacts system ranking validity.

Method

The authors used a discourse-aware evaluation framework combined with human auditing to identify and categorize five types of annotation errors in three long-form summary benchmarks.

In practice

• Audit existing faithfulness benchmarks for discourse inconsistencies.
• Incorporate discourse-aware checks into annotation guidelines.
• Re-evaluate system rankings using revised faithfulness labels.

Topics

• Faithfulness Annotation
• Long-form Summarization
• Language Model Evaluation
• Benchmark Reliability
• Discourse Analysis
• Annotation Error Taxonomy

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling — NL-to-FOL benchmarks contain significant errors, distorting model evaluation, but LLM-assisted review can efficiently correct them.
• Faithful by Construction: Claim-Anchored Attribution for Multi-Document Summarization — A modular, claim-anchored summarization framework inherently builds fine-grained attribution and faithfulness into the generation process.
• Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation — A new framework evaluates LLM summaries of parliamentary debates by preserving argumentative reasoning.
• Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding — ConSUM enhances summary factuality by combining source consistency with candidate consensus via MBR reranking.
• G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment — Cross-lingual idiom alignment challenges LLMs due to non-compositionality, with glosses offering semantic grounding but literal bias remaining dominant.
• A Resource for Enthymeme Detection in Controversial Political Discourse — The study of annotator disagreement in enthymeme detection improves model performance and understanding of human inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.