Cracks in the Bridge—or A Bridge Too Far? Comparing Human and LLM Errors in the Annotation of Bridging Anaphora

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, quick

Summary

An error analysis was conducted on human and LLM annotation data from the GUMBridge corpus, focusing on varieties of bridging anaphora. The study explored the distribution of precision and recall errors made by annotators and their correlation with bridging subtypes. Findings indicate that while LLMs perform substantially worse than human annotators overall, they exhibit more balanced precision and recall scores compared to humans, whose performance strongly favors precision. Regarding subtypes, comparison and meronomy relations were found to be easier to reliably annotate than the more broadly construed entity relations for both human and LLM annotators. However, LLM errors were more broadly distributed across subtypes than human errors.

Key takeaway

For NLP Engineers building annotation pipelines for bridging anaphora, consider the trade-offs between human precision and LLM recall balance. While LLMs generally underperform, their more balanced error distribution might be advantageous for tasks where missing connections (low recall) is more detrimental than incorrect ones. Conversely, leverage human annotators for high-precision requirements, especially for complex entity relations. Tailor your annotation strategy by subtype, assigning easier meronomy or comparison relations to LLMs if overall accuracy is less critical than throughput.

Key insights

LLMs underperform humans in bridging anaphora annotation but show more balanced precision/recall, with specific subtype annotation difficulties.

Principles

• Human annotators prioritize precision in anaphora.
• LLM annotation errors are broadly distributed.
• Comparison and meronomy are easier bridging subtypes.

Method

The paper performs an error analysis on human and LLM annotation data from the GUMBridge corpus, exploring precision/recall distribution and correlation with bridging subtypes.

In practice

• Consider LLM balance for recall-sensitive tasks.
• Focus human annotation on complex entity relations.
• Prioritize meronomy/comparison for initial LLM tasks.

Topics

• Bridging Anaphora
• LLM Annotation
• Error Analysis
• GUMBridge Corpus
• Natural Language Processing
• Corpus Annotation

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling — NL-to-FOL benchmarks contain significant errors, distorting model evaluation, but LLM-assisted review can efficiently correct them.
• Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection — LLM labels can match human-level F1 scores for hostility detection but exhibit distinct error profiles.
• TalkTag: Fine-Grained Morphosyntactic Error Annotation for Transcribed Speech — TalkTag automates fine-grained morphosyntactic error annotation in spoken transcripts using an LLM, even with scarce data.
• Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing — LLMs struggle with deep metaphor comprehension, often misinterpreting concepts and relying on lexical associations over context or full syntactic understanding.
• Creation of the Estonian Subjectivity Dataset: Assessing the Degree of Subjectivity on a Scale — Human subjectivity annotation is inherently variable and influenced by context, while LLMs offer plausible but distinct scoring.
• The Data Problem in Low-Resource Languages — Data availability, quality, and standardization are critical for NLP model performance across diverse languages.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.