The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers

2026-06-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Research Practices & Reproducibility · Depth: Advanced, quick

Summary

An analysis of 56,800 conference papers published at five leading AI venues between 2014 and 2024 reveals a substantial improvement in documentation practices within the AI research community. The study identified seven reproducibility variables, finding that papers sharing both code and data increased nearly sixfold, from 11% in 2014 to 64% in 2024. This trend suggests a broader movement toward open science, as these improvements largely predate the introduction of formal reproducibility checklists by impactful venues. Based on these enhanced documentation practices, the analysis estimates that the overall reproducibility of AI research, inferred rather than directly tested, rose from 28% in 2014 to 64% in 2024, addressing the ongoing reproducibility crisis.

Key takeaway

For AI Research Scientists evaluating publication strategies or assessing research quality, this analysis confirms a strong community-driven shift towards open science. You should prioritize sharing both code and data in your publications, as this practice is now a dominant expectation, increasing from 11% to 64% of papers over the last decade. Embracing these documentation practices will align your work with evolving community norms and significantly enhance the perceived reproducibility of your research.

Key insights

AI research documentation practices improved significantly from 2014-2024, driven by a broader open science movement.

Principles

• Open science drives reproducibility.
• Documentation correlates with reproducibility.
• Community shifts precede formal mandates.

Method

Analyzed 56,800 papers from five leading AI conferences (2014-2024) using seven reproducibility variables to track documentation changes.

Topics

• AI Research
• Open Science
• Reproducibility
• Documentation Practices
• Conference Papers
• Code Sharing
• Data Sharing

Best for: AI Scientist, Research Scientist

Related on AIssential

• Beyond the PDF: Rowan Cockett on Reproducible, Composable Science — Reproducibility in science requires integrating data, code, and narrative through open standards and modular systems.
• The uncritical adoption of AI in science is alarming — we urgently need guard rails — Uncritical AI adoption in science risks reduced research quality, narrowed scope, and deskilling future scientists.
• ICLR’s Commitment to OpenReview — ICLR is doubling its financial support for OpenReview to promote open science and reproducible research.
• Struggling to reproduce paper results before improving them — stuck below reported accuracy [R] — Reproducibility gaps are common in ML research due to missing details or undisclosed methods.
• Farewell Ai2 — Open science and public scientists are crucial for AI's safe diffusion, diverse ecosystem, and future talent development, counteracting the trend of closed industry research.
• 142 - Science Of Science, with Kyle Lo — SciSci uses computational methods to study scientific processes, guiding tool development for community-wide benefit.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.