Experimenting with AI in a Living Literature Review
Summary
The AI Accountability Review (AIAR) is a living literature review that tracks AI accountability literature for researchers and policymakers. This article explores how AI, specifically OpenAI's Agent mode, InoReader's LLM integration, Google Colab, and Google's NotebookLM with Gemini, can assist in the literature review process. The author details experiments across four phases: search, appraisal, synthesis, and interpretation, as outlined by Fok et al. (2025). Key successes include automating the scraping and formatting of conference proceedings into RSS feeds, gathering author email addresses for promotion, and using LLMs for second opinions on article relevance and updating existing articles. While AI shows promise for descriptive tasks and automation, its utility for complex synthesis and interpretation, which require expert framing and contextualization, remains limited.
Key takeaway
For research scientists managing living literature reviews, you should integrate AI tools like OpenAI's Agent mode and LLM-powered appraisal systems to automate data collection and initial relevance checks. This frees up time for the critical, human-centric tasks of synthesis, interpretation, and contextualization, where your expert judgment is irreplaceable. Always double-check AI-generated content for accuracy and comprehensiveness.
Key insights
AI can automate literature review tasks like scraping and appraisal, but human expertise remains crucial for synthesis and interpretation.
Principles
- AI excels at automation with oversight.
- Human oversight is critical for AI outputs.
- Framing and context require expert judgment.
Method
Utilize OpenAI's Agent mode for scraping and formatting, InoReader for LLM-powered appraisal, and NotebookLM with Gemini for grounded synthesis and evidence checking.
In practice
- Use Agent mode to collect and format conference papers.
- Employ LLMs for second opinions on article relevance.
- Verify claims with NotebookLM's grounded synthesis.
Topics
- AI-Assisted Literature Review
- Large Language Models
- Information Extraction Automation
- Research Synthesis
- AI Accountability
Best for: AI Researcher, Policy Maker, Research Scientist
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by AI Accountability Review.