Scientists invented a fake disease. AI told people it was real
Summary
Scientists conducted an experiment where they created two fabricated research papers, explicitly stating "This is entirely made up" within the results section, and uploaded them to a preprint server. Subsequently, artificial intelligence models began incorporating these fake papers and the invented disease as real entities in their outputs. More concerningly, these fabricated papers were later cited in legitimate, peer-reviewed scientific literature, indicating a broader issue within academic vetting processes. This experiment highlights a vulnerability in how AI systems process and validate information, particularly when exposed to misinformation, and also points to a breakdown in human oversight within scientific citation and peer review.
Key takeaway
For AI/ML teams developing information retrieval or summarization systems, your data ingestion pipelines must incorporate sophisticated plausibility checking and real-time Bayesian updating. You should prioritize mechanisms that flag or filter content explicitly labeled as fabricated, even if it appears in academic contexts. This will mitigate the risk of your models inadvertently propagating misinformation and eroding user trust.
Key insights
AI models can propagate misinformation, even when explicitly labeled as fake, highlighting data validation challenges.
Principles
- AI systems lack inherent common sense for truth validation.
- Reliance on single information sources is inherently risky.
In practice
- Implement robust data provenance checks for AI training.
- Cross-reference AI-generated information with multiple sources.
Topics
- AI Misinformation
- Scientific Integrity
- Preprint Servers
- Academic Citation
- Large Language Models
Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, Research Scientist, AI Ethicist
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.