Gold Label Errors in the SciFact Benchmark: An LLM-Assisted Annotation Audit

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, quick

Summary

A systematic annotation audit of the SciFact benchmark, a widely-used scientific claim verification dataset with 645 citations, identified significant gold-label errors. Researchers conducted the first systematic audit of its development and training sets, combining automated screening with a small language model (costing \$0.11 in API fees) and exhaustive manual verification. They found 11 errors (5.3%, 95% CI 2.7–9.2%) in 209 audited development set claim-document pairs, with 9 of these incorrectly labeled as SUPPORT. Additionally, 13 errors (2.3%, 95% CI 1.2–3.9%) were found in 564 training set pairs. Correcting these development labels improved binary macro-F1 scores by 1.7–3.8 points for models like GPT-5.4 and Claude Haiku 4.5, with 3-way evaluation gains up to +9.2 points. Corrected annotations and an annotator packet have been released, recommending their use.

Key takeaway

For NLP Engineers and Research Scientists working on scientific claim verification, you should immediately update your SciFact benchmark usage. The identified gold-label errors significantly skew model performance metrics, with corrections improving F1 scores by 1.7–3.8 points. Using the newly released corrected annotations will ensure more accurate and reliable evaluation of your models, preventing misleading performance assessments and fostering more robust research outcomes.

Key insights

Gold-label errors in widely-used benchmarks significantly impact model performance and require systematic, LLM-assisted audits.

Principles

• Benchmark quality directly affects model evaluation.
• LLM-assisted audits are cost-effective for large datasets.
• Annotation errors can exhibit directional asymmetry.

Method

Combine automated screening with a small language model and exhaustive manual verification against source publications to audit benchmark annotations.

In practice

• Audit existing benchmarks before deployment.
• Use LLMs for initial error screening.
• Recast mislabeled evidence as NEI for improved evaluation.

Topics

• Scientific Claim Verification
• SciFact Benchmark
• Annotation Audit
• Large Language Models
• Benchmark Evaluation
• Gold-Label Errors

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling — NL-to-FOL benchmarks contain significant errors, distorting model evaluation, but LLM-assisted review can efficiently correct them.
• CommunityFact: A Dynamic, Multilingual, Multi-domain Benchmark for Misinformation Detection in the Wild — CommunityFact offers a dynamic benchmark revealing LLM misinformation detection challenges and web-search benefits.
• The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection — Current statistical methods for detecting LLM benchmark contamination are unreliable in realistic auditing due to distribution shift and scale.
• HellaSwag or HellaBad? 36% of this popular LLM benchmark contains errors — Academic LLM benchmarks often misalign with real-world performance and contain significant data quality issues.
• FinBalance: A Multi-Document Accounting Reconciliation Benchmark — LLMs struggle with multi-document accounting reconciliation, showing low accuracy and issues with document binding and consistent aggregation.
• PyMETA: A Benchmark Dataset for Hierarchical Student Code Error Classification with Python-Interpreter-Based Labels — PyMETA offers a large, hierarchically-taxonomized Python code error dataset for LLM evaluation, revealing biases and performance gaps.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.