Recognizing reproducibility and reusability in times of fast science

2026-03-25 · Source: Nature Machine Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Advanced, medium

Summary

Nature Machine Intelligence emphasizes the critical role of reproducibility and reusability in scientific research, especially with the rapid adoption of large language models (LLMs). The journal promotes transparent code sharing and reporting through initiatives like Reusability Reports, an article format introduced in 2020 that evaluates the robustness and extensibility of published code. Building on existing guidelines like FAIR and FAIR4RS principles, new concepts of "reviewability" and "supportability" are proposed to enhance peer review of research software, focusing on clarity, legibility, and explicit evidence chains linking code to results. To aid this, authors are encouraged to use executable compute capsules via CodeOcean, allowing reviewers to run and test code in a ready-made environment. The increasing use of LLMs, with their non-deterministic outputs and frequent updates, presents new challenges for reproducibility, necessitating clear documentation of model versions, prompts, and outputs.

Key takeaway

For research scientists developing computational models, ensuring your code is highly reviewable and supportable is crucial. You should structure your code repositories logically, include a detailed ReadMe, and consider using platforms like CodeOcean for executable compute capsules. This approach facilitates peer review and strengthens the reproducibility of your findings, particularly when incorporating non-deterministic elements like large language models.

Key insights

Reproducibility and reviewability are paramount for scientific integrity, especially with LLM integration.

Principles

• Code must be reviewable and supportable.
• FAIR principles extend to research software.
• LLM usage requires clear documentation.

Method

Authors can create executable compute capsules on platforms like CodeOcean, providing a ready-made environment for reviewers to run and test code and data.

In practice

• Use CodeOcean for executable compute capsules.
• Provide comprehensive ReadMe files.
• Document LLM versions and prompts.

Topics

• Reproducibility
• Research Software
• Large Language Models
• Peer Review
• Code Reusability

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• Beyond the PDF: Rowan Cockett on Reproducible, Composable Science — Reproducibility in science requires integrating data, code, and narrative through open standards and modular systems.
• Automated reproducibility assessments in the social and behavioral sciences using large language models — Large language models can automate reproducibility assessments in social sciences, outperforming human reanalysts in qualitative agreement.
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues — ReproRepo scales reproducibility audits using human-raised GitHub issues as supervision for LLM agents.
• The Notebook That Science Built Its Future On Is Broken. Here’s the Proof — and the Fix. — Jupyter notebooks, despite widespread use, pose a severe reproducibility challenge for published scientific research.
• It's not the Language Model, it's the Tool: Deterministic Mediation for Scientific Workflows — Typed mediation uses LLMs to orchestrate deterministic, researcher-defined tools for reproducible scientific analysis, ensuring consistent results.
• [D] ICML paper to review is fully AI generated — AI-generated research papers challenge conference policies and reviewer assessment of technical merit and authorship.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Nature Machine Intelligence.