AISysRev -- LLM-based Tool for Title-abstract Screening

2026-04-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Intermediate, long

Summary

AiSysRev is a new LLM-based web application designed to accelerate the title-abstract screening phase of systematic reviews, a notoriously laborious task in software engineering that can take 67 weeks on average. Implemented as a Docker container, the tool accepts CSV files of paper titles and abstracts, allowing users to define inclusion/exclusion criteria. It supports multiple LLMs via OpenRouter, offering both zero-shot and few-shot screening capabilities. AiSysRev also integrates manual screening, displaying LLM outputs as guidance for human reviewers, and allows exporting results to CSV for further analysis. A trial study with 137 papers identified four classification categories: Easy Includes, Easy Excludes, Boundary Includes, and Boundary Excludes, highlighting that human intervention remains crucial for "Boundary" cases where LLMs are prone to errors. The tool aims to reduce the burden of assessing large volumes of scientific literature, particularly for rapid reviews.

Key takeaway

For AI Engineers and Research Scientists conducting systematic literature reviews, AiSysRev offers a practical approach to significantly reduce screening effort. You should integrate this Dockerized tool to automate initial title-abstract screening, especially for high-volume or rapid reviews. While LLMs handle clear inclusions/exclusions efficiently, always plan for human review of "Boundary" cases, leveraging the tool's probability scores to focus your efforts where LLMs are most likely to err, ensuring accuracy without sacrificing speed.

Key insights

LLMs can significantly expedite systematic review screening, but human oversight remains critical for ambiguous cases.

Principles

• LLMs excel at clear-cut screening decisions.
• Human judgment is essential for "Boundary" cases.
• Few-shot prompting improves LLM screening performance.

Method

AiSysRev processes CSV input with titles/abstracts, applies user-defined criteria via multiple LLMs (zero-shot/few-shot), and provides LLM outputs for guided manual review, exporting results to CSV.

In practice

• Use multiple LLMs for screening to compare results.
• Prioritize manual review for papers with uncertain LLM scores.
• Apply cutoff rules in rapid reviews for efficiency.

Topics

• AISysRev Tool
• LLM-based Screening
• Systematic Literature Reviews
• Title-Abstract Screening
• Software Engineering Research

Code references

• EvoTestOps/AISysRev
• EvoTestOps/AiSysRev

Best for: Research Scientist, AI Scientist, AI Engineer

Related on AIssential

• Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations — LLMs in systematic review screening fail predictably due to semantic and lexical issues, requiring specific mitigation strategies.
• Meet an AI Using Scholar — AI tools show promise for systematic review screening and descriptive data extraction, but struggle with comprehensive search and nuanced data interpretation.
• Automated reproducibility assessments in the social and behavioral sciences using large language models — Large language models can automate scientific reproducibility checks, matching or exceeding human reanalysis performance.
• Towards LLM Accelerated Rapid Reviews for Software Tool Discovery -- Case for Log Anomaly Detection — LLMs can significantly accelerate software tool discovery and validation in rapid reviews, reducing human effort.
• Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio — LLM agents critically fail at criterion-based screening in scientific meta-analysis, despite high retrieval recall.
• Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes — An LLM-based system can accurately predict peer-review outcomes and provide structured feedback for academic manuscripts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.