Why AI can’t be trusted to write scientific reviews

2026-05-26 · Source: Machine learning : nature.com subject feeds · Field: Science & Research — Health & Medical Research, Research Methodology & Innovation, Artificial Intelligence & Machine Learning · Depth: Intermediate, quick

Summary

The Cochrane Collaboration, a London-based publisher specializing in high-quality health-related systematic reviews, is evaluating artificial-intelligence tools for increasing review efficiency and scale. Despite the potential, current AI tools are deemed unready for mainstream adoption in this high-stakes field, where errors could impact clinical practice and public-health policy. While AI models mimic human review processes like study identification and data extraction, they struggle with defining meaningful questions, interpreting results, and understanding clinical implications due to a lack of context, subjective nuance, and a tendency to hallucinate. Furthermore, most available tools are proprietary "black box" systems from private companies, posing independence issues for reviews of drugs and medical devices. Practical experience shows these tools require extensive training and currently make the review process longer than manual methods.

Key takeaway

For research scientists or AI ethicists considering AI integration into systematic reviews, recognize that current tools are not a direct replacement for human expertise. Your focus should shift from full automation to designing collaborative workflows where AI supports, rather than dictates, critical tasks. Prioritize tools with transparency and open-source models to mitigate risks of bias and ensure independence, especially in health-related evidence synthesis.

Key insights

Current AI tools are not ready for high-stakes scientific systematic reviews due to limitations in context, nuance, and transparency.

Principles

• Human specialists are essential for defining review questions, interpreting results, and understanding implications.
• Proprietary "black box" AI tools introduce independence and bias risks in sensitive scientific reviews.
• AI's optimal role in scientific review is collaborative, augmenting human expertise rather than replacing it.

Method

Developers should build systems enabling effective human-AI collaboration for study assessment, moving beyond AI generating individual reviews.

In practice

• Expect current AI tools for screening and data extraction to potentially increase overall review time.
• Scrutinize AI tool provenance and transparency, particularly for reviews evaluating drugs or medical devices.

Topics

• AI in Scientific Review
• Systematic Reviews
• Health Policy
• AI Hallucination
• Human-AI Collaboration
• Proprietary AI

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Research Scientist, AI Scientist, AI Ethicist

Related on AIssential

• Meet an AI Using Scholar — AI tools show promise for systematic review screening and descriptive data extraction, but struggle with comprehensive search and nuanced data interpretation.
• What happens when scientists trust AI more than colleagues? — Rapid AI adoption in science risks eroding critical thinking and human relationships essential for rigorous research.
• Welcoming AI as a New Colleague: How Should We Evaluate AI for Science? — AI's rapid integration into science necessitates robust evaluation frameworks, emphasizing human-AI collaboration to leverage complementary strengths.
• How to build an AI Scientist: first peer-reviewed paper spills the secrets — AI Scientist, an autonomous research tool, successfully generated a peer-reviewed paper, demonstrating AI's capacity for scientific discovery.
• The social sciences need tools for the 21st century | Letters — Social science needs new tools and methods to improve replicability and advance understanding of human behavior.
• Experimenting with AI in a Living Literature Review — AI can automate literature review tasks like scraping and appraisal, but human expertise remains crucial for synthesis and interpretation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine learning : nature.com subject feeds.