Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations

2026-06-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, quick

Summary

A study investigated the reliability of large language models (LLMs) for title-abstract screening in systematic reviews (SRs), moving beyond quantitative agreement metrics to understand failure causes. Researchers analyzed disagreements between LLMs and human experts across six software engineering SRs, involving over 1,000 primary study papers. LLMs operated in zero-shot mode, yielding Kappa values from 0.52 to 0.77. Qualitative analysis revealed recurring disagreement causes, including boundary ambiguity in key terms, keyword overemphasization, and incorrect topic inference. Based on these findings, the study proposes recommendations such as validating LLM semantic understanding pre-deployment, employing multiple LLMs, and concentrating validation efforts on borderline screening cases.

Key takeaway

For MLOps Engineers deploying LLMs for systematic review screening, your focus should extend beyond aggregate agreement metrics. Instead of solely relying on Kappa values, you must qualitatively investigate disagreement causes like semantic ambiguity and keyword overemphasis. Implement pre-deployment semantic validation, consider running multiple LLMs, and prioritize human review for borderline cases to enhance screening reliability and reduce costly errors.

Key insights

LLM failures in systematic review screening are rooted in identifiable semantic and inference issues, not merely quantitative disagreement.

Principles

• LLM-human screening disagreements have identifiable causes.
• Semantic ambiguity and keyword overemphasis hinder LLM accuracy.
• Incorrect topic inference is a common LLM screening failure.

Method

Disagreements between zero-shot LLMs and human experts were qualitatively analyzed across six software engineering systematic reviews, involving over 1,000 papers, to identify failure causes.

In practice

• Validate LLM semantic understanding pre-deployment.
• Employ multiple LLMs for screening tasks.
• Prioritize validation on borderline screening cases.

Topics

• Large Language Models
• Systematic Reviews
• Title-Abstract Screening
• LLM Reliability
• Disagreement Analysis
• Zero-shot Learning

Best for: Research Scientist, AI Scientist, MLOps Engineer

Related on AIssential

• Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations — LLMs in systematic review screening fail predictably due to semantic and lexical issues, requiring specific mitigation strategies.
• Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio — LLM agents face a critical bottleneck in screening eligible studies for meta-analysis, failing to reliably distinguish relevant from irrelevant content.
• From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text — Evaluating LLMs for legal text requires both quantitative benchmarks and qualitative error analysis to reveal true capabilities.
• When Evidence Conflicts: Uncertainty and Order Effects in Retrieval-Augmented Biomedical Question Answering — Conflicting evidence and document order significantly degrade LLM accuracy and calibration in biomedical QA.
• On the Limits of LLM-as-Judge for Scientific Novelty Assessment — LLMs are unreliable judges of scientific novelty for research questions, often creating a "novelty mirage."
• Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options — Scaling multiple-choice evaluations to 100 options reveals LLM competence gaps obscured by low-option benchmarks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.