Scaling and quantization of a foundational deep learning model for network biology

2026-05-11 · Source: Machine learning : nature.com subject feeds · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Life Sciences & Biology · Depth: Expert, quick

Summary

A Geneformer model was developed using an expanded pretraining dataset comprising over 100 million single-cell human transcriptomes. This significant increase in data diversity and model size led to improved downstream biological predictions, particularly in network biology applications. To enhance accessibility and reduce computational demands, model quantization was implemented. This technique successfully preserved the model's acquired biological knowledge while substantially decreasing the required GPU compute resources. The original Geneformer model was reported in Nature in 2023, and the quantization method applied here is based on the QLoRA technique from 2023. This work, published in Nat. Comput. Sci. on May 11, 2026, focuses on scaling and quantization for resource-efficient predictions.

Key takeaway

For AI Engineers working with large biological foundation models, you should consider implementing model quantization techniques, such as those derived from QLoRA, to significantly reduce GPU compute requirements. This approach allows you to maintain high predictive performance in areas like network biology while making your models more accessible and cost-effective to deploy and fine-tune on more modest hardware configurations.

Key insights

Expanding Geneformer with more data and quantization improves biological prediction while reducing GPU demands.

Principles

• Data diversity enhances predictive performance.
• Quantization preserves knowledge, reduces compute.

Method

The Geneformer model was expanded with >100 million human transcriptomes, then quantized using a method similar to QLoRA to reduce GPU requirements while maintaining biological predictive power.

In practice

• Use large-scale transcriptomic datasets for pretraining.
• Apply model quantization to reduce GPU memory.
• Improve model accessibility via compute reduction.

Topics

• Geneformer Model
• Network Biology
• Deep Learning
• Model Quantization
• Single-cell Transcriptomes

Best for: AI Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Optimizing AI Models with Quanto on H100 GPUs — Quanto optimizes AI models by quantizing parameters to lower precision, reducing memory and speeding inference.
• Evaluating the role of pretraining dataset size and diversity on single-cell foundation model performance — Single-cell foundation models do not show clear data scaling laws, unlike large language models.
• Retentive Network promotes efficient RNA language modeling of long sequences — RNAret, a Retention Network-based model, efficiently processes long RNA sequences with O(n) complexity, outperforming O(n²) Transformers.
• How a 2021 Quantization Algorithm Quietly Outperforms Its 2026 Successor — Optimal analytical scaling in vector quantization significantly improves accuracy and efficiency over fixed scaling.
• QuickReduce FP4 Quantization and Benchmarking on MI355 — QuickReduce with FP4/INT4 quantization on MI355 significantly accelerates large-message all-reduce operations for LLM inference.
• Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models — For embedding-heavy models, communication, not compute, dominates multi-node scaling, requiring direct optimization of data transfer.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine learning : nature.com subject feeds.