An Empirical Investigation of Pre-Trained Deep Learning Model Reuse in the Scientific Process

2024-07-02 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Advanced, extended

Summary

An empirical study investigated the reuse of pre-trained deep learning models (PTMs) in natural sciences, analyzing 17,511 peer-reviewed, open-access papers from January 1st, 2000, to December 10th, 2025, using an automated large language model pipeline. The research identified 631 papers reusing PTMs, finding that "Biochemistry, Genetics and Molecular Biology" leads other fields in adoption. "Adaptation" reuse, involving fine-tuning existing models, was the most prevalent pattern, accounting for 70.29% of instances. The study also revealed that PTM integration primarily impacts the "Test" stage of the scientific process (422 instances), with models like AlexNet (25), Alphafold 2 (23), VGG 16 (19), ResNet 50 (18), and Alphafold (18) being most frequently reused. This work characterizes current PTM reuse practices and their impact on scientific workflows.

Key takeaway

For research scientists and AI engineers integrating deep learning into scientific discovery, you should prioritize pre-trained models, especially through "adaptation" reuse, to reduce computational overhead and accelerate data-driven insights. While PTMs currently excel in the "Test" phase, consider expanding their application to earlier stages like hypothesis generation or experimental design to unlock their full potential across the entire scientific workflow.

Key insights

Pre-trained model reuse, particularly adaptation, is increasingly common in natural sciences, predominantly augmenting the "Test" stage.

Principles

• PTMs significantly reduce deep learning training costs.
• "Adaptation" is the dominant PTM reuse pattern in natural sciences.
• PTMs function as reusable scientific infrastructure.

Method

An automated LLM-driven pipeline identifies deep learning usage, verifies PTM reuse, extracts specific PTMs, classifies reuse patterns (conceptual, adaptation, deployment), and maps their impact to scientific process stages.

In practice

• Adopt PTMs to mitigate deep learning computational costs.
• Focus on adaptation for domain-specific scientific tasks.
• Explore PTM integration in earlier scientific stages beyond testing.

Topics

• Deep Learning Model Reuse
• Pre-trained Models
• Scientific Workflows
• Empirical Software Engineering
• AI for Science
• Automated Literature Review
• Biochemistry

Code references

• jax-ml/jax
• tensorflow/models
• jgm/pandoc
• huggingface/diffusers

Best for: Research Scientist, AI Scientist, Software Engineer

Related on AIssential

• Next Stage of AI Scientist: NanoResearch (Skills, Mem, RL) — NanoResearch enables personalized AI scientific discovery through co-evolution with human researchers, adapting to individual styles.
• When Deep Learning Meets Physics — Integrating physical laws into neural networks is a crucial, under-recognized advancement for AI's real-world utility.
• It's not the Language Model, it's the Tool: Deterministic Mediation for Scientific Workflows — Deterministic tools orchestrated by language models ensure reproducibility in scientific workflows, unlike direct code generation.
• SciPaths: Forecasting Pathways to Scientific Discovery — Forecasting scientific discovery requires identifying specific enabling contributions and their prior-work groundings, a task current LLMs struggle with.
• ADAPTive Input Training for Many-to-One Pre-Training on Time-Series Classification — ADAPT enables mixed-batch pre-training for time-series data by aligning physical properties across diverse datasets.
• Adaptation: The Missing Layer Between Apps and Foundation Models — Gradient-free, inference-time adaptation bridges enterprise AI's last-mile reliability gap more efficiently than scaling or fine-tuning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.