From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence

2025-10-13 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

PrimeFacts is a new methodology and resource designed to extract fine-grained, structured evidence from full fact-checking articles, making it accessible for automated verification systems. It comprises 13,106 PolitiFact articles, each with claims, verdicts, and referenced sources, identifying 49,718 in-article hyperlinks as key evidence anchors. The framework employs large language models (LLMs) to rewrite these anchor sentences into context-independent premises, and also explores implicit evidence extraction. Evaluations demonstrate significant performance gains: decontextualized evidence improves Mean Reciprocal Rank by up to 30% for cross-article evidence retrieval and raises Macro-F1 by 10-20 points for verdict prediction across 2-class and 5-class granularities. These improvements are consistent across various LLM architectures, including Qwen3 and Llama models.

Key takeaway

For NLP Engineers building automated fact-checking systems, you should integrate decontextualized evidence extraction into your workflows. Leveraging LLMs to transform hyperlink-anchored sentences into self-contained premises significantly boosts evidence retrieval and claim verification accuracy. Consider a hybrid approach, combining anchor-based extraction with open LLM generation, to maximize evidence coverage and system robustness. This strategy enhances transparency and reusability of journalistic fact-checks.

Key insights

Decontextualizing hyperlink-anchored evidence from fact-checks significantly improves automated retrieval and verification performance.

Principles

• In-article hyperlinks reliably signal core evidence.
• Decontextualized evidence enhances portability.
• Evidence reuse extends claim matching.

Method

PrimeFacts extracts evidence using three modes: verbatim anchored sentences, LLM-based decontextualization, and open LLM extraction. It defines an evidence-type ontology and evaluates faithfulness with DFS.

In practice

• Use LLMs to decontextualize evidence sentences.
• Anchor evidence extraction to in-article hyperlinks.
• Combine anchor-based and open extraction for coverage.

Topics

• Fact-Checking Automation
• Large Language Models
• Information Extraction
• Evidence Decontextualization
• PrimeFacts Resource
• PolitiFact Dataset

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Explicit Evidence Grounding via Structured Inline Citation Generation — LLMs excel at document retrieval but struggle with precise evidence span identification for faithful inline citations.
• CREDENCE: Claim Reduction for Decomposition & Enhanced Credibility -- Semantic Metrics and Convergence Analysis — Semantic-F1 and formal repair analysis improve claim decomposition for reliable automated fact-checking.
• Do Current Retrievers Cover All the Evidence? A Controlled Study of Conjunctive Cross-Page Retrieval — Current retrieval systems struggle to ensure all conjunctive evidence is covered across document pages, even when relevant documents are found.
• CRAFT: A Unified Counterfactual Reasoning Framework for Tabular Question Answering and Fact Verification — Counterfactual reasoning significantly improves LLM performance and robustness in complex tabular QA and fact verification.
• Discovery of Legal Patterns in Civil Petitions via LLM-Based Fact Extraction and Density Clustering — LLM-based fact extraction significantly improves legal document clustering by reducing noise and enhancing semantic coherence.
• VaxScope: Document-Level Structured Evidence Extraction from Immunization Systematic Reviews — Document-level structured evidence extraction from systematic reviews requires aggregating distributed information beyond isolated text spans.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.