Chaptered Recording: The Past, Present, and Futures of Pragmatic and Responsible Adoption of AI in…

2026-07-12 · Source: AI on Medium · Field: Science & Research — Research Methodology & Innovation, Artificial Intelligence & Machine Learning, Health & Medical Research · Depth: Intermediate, short

Summary

Farhad Shokraneh's chaptered recording outlines the pragmatic and responsible adoption of AI in systematic reviews, spanning a timeline from 1991 to 2026. The lecture classifies AI types used, including rule-based, machine learning, predictive, discriminative, generative AI, LLMs, and agentic AI, detailing their application across systematic review steps. It covers protocol development, various search classifications, screening techniques like active learning and classifiers (e.g., in Rayyan, Covidence, ASReview, EPPI-Reviewer), and data extraction, including from images and tables using tools like Elicit, Nested Knowledge, and PICO Portal. The presentation also addresses prompt engineering, factors affecting LLM performance, reported issues, and methods for preventing/detecting LLM errors. Ethical considerations, such as journal policies on AI use and copyright implications for paywalled PDFs with LLMs, are discussed, alongside the "Golden Rules" for using, evaluating, and developing new AI tools in this domain.

Key takeaway

For research scientists conducting systematic reviews, the integration of AI and LLMs presents both efficiency gains and critical challenges. You should carefully evaluate AI tools like Elicit or Covidence for specific steps, understanding their limitations in data extraction and screening. Prioritize tools that offer explainability and adhere to ethical guidelines, especially regarding copyrighted material. Develop robust prompt engineering strategies and consider Safety RAG to minimize hallucination and ensure data integrity in your research.

Key insights

AI, particularly LLMs, offers comprehensive automation for systematic reviews, requiring pragmatic adoption and careful error management across all stages.

Principles

• Empirically test AI tools for task suitability and risks.
• Explainability (SIB) is crucial for trust in AI outputs.
• Adhere to copyright and ethical guidelines for AI use.

Method

Prompt engineering involves developing and testing prompts for LLMs. Safety RAG combines Retrieval-Augmented Generation with Knowledge Graphs to enhance reliability and reduce hallucination in AI-assisted systematic reviews.

In practice

• Apply active learning for efficient screening relevancy ranking.
• Implement prompt engineering for LLM-driven review tasks.
• Use Safety RAG to mitigate LLM hallucination in searches.

Topics

• Systematic Reviews
• AI Automation
• Large Language Models
• Prompt Engineering
• Active Learning
• Evidence Synthesis
• AI Ethics

Best for: Research Scientist, AI Scientist, AI Ethicist

Related on AIssential

• Meet an AI Using Scholar — AI tools show promise for systematic review screening and descriptive data extraction, but struggle with comprehensive search and nuanced data interpretation.
• What Does It Take to Research with AI? A Rapid Review of Competencies to Train LLM-Literate Researchers — LLM-assisted research requires integrated human competencies, prioritizing accountability, critical judgment, and ethical oversight over mere technical skills.
• Towards LLM Accelerated Rapid Reviews for Software Tool Discovery -- Case for Log Anomaly Detection — LLMs can significantly accelerate software tool discovery and validation in rapid reviews, reducing human effort.
• Automated reproducibility assessments in the social and behavioral sciences using large language models — Large language models can automate scientific reproducibility checks, matching or exceeding human reanalysis performance.
• Why AI can’t be trusted to write scientific reviews — Current AI tools are not ready for high-stakes scientific systematic reviews due to limitations in context, nuance, and transparency.
• Experimenting with AI in a Living Literature Review — AI can automate literature review tasks like scraping and appraisal, but human expertise remains crucial for synthesis and interpretation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.