Retentive Network promotes efficient RNA language modeling of long sequences

2026-03-11 · Source: Machine learning : nature.com subject feeds · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Life Sciences & Biology, Data Science & Analytics · Depth: Expert, long

Summary

RNAret, an RNA language model based on the Retention Network, has been developed to address the O(n²) complexity limitations of Transformer-based models in processing long RNA sequences. Published on March 11, 2026, this model achieves O(n) complexity, enabling efficient training parallelism, reduced computational overhead, and improved long-sequence processing. RNAret was pretrained using a self-supervised masked language modeling approach on 29.8 million RNA sequences. Experimental results demonstrate its superior performance across various tasks, including RNA-RNA interaction prediction, RNA secondary structure prediction, and mRNA/lncRNA classification, highlighting its potential for extracting latent features and advancing RNA biology understanding. The source code, pretraining scripts, and model weights are publicly available on GitHub and Zenodo, with a web server also provided.

Key takeaway

For research scientists working with long RNA sequences, RNAret offers a significant advantage over traditional Transformer models due to its O(n) complexity. You should consider integrating RNAret into your computational biology workflows to improve efficiency and accuracy in tasks like RNA-RNA interaction and secondary structure prediction, leveraging its publicly available code and pretrained weights for immediate application.

Key insights

RNAret, a Retention Network-based model, efficiently processes long RNA sequences with O(n) complexity, outperforming O(n²) Transformers.

Principles

• Retention Networks offer O(n) complexity for long sequences.
• Self-supervised masked language modeling is effective for RNA pretraining.

Method

RNAret employs a retention mechanism for O(n) complexity, enabling training parallelism and low computational overhead. It is pretrained via self-supervised masked language modeling on large RNA datasets.

In practice

• Utilize RNAret for RNA-RNA interaction prediction.
• Apply RNAret to RNA secondary structure prediction tasks.
• Use RNAret for mRNA/lncRNA classification.

Topics

• RNA Language Modeling
• Retention Network
• Transformer Models
• RNA Secondary Structure Prediction
• Self-supervised Learning

Code references

• DrBlackZJU/RNAret

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• Scaling and quantization of a foundational deep learning model for network biology — Expanding Geneformer with more data and quantization improves biological prediction while reducing GPU demands.
• Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes — DID improves diffusion language models by replacing masking with deletion-insertion for efficiency and flexibility.
• An Algebraic View of the Expressivity of Recurrent Language Models — Arithmetic semantics fundamentally determines recurrent neural network expressivity and recognized formal languages.
• Predicting RNA 3D structure and conformers using a pre-trained secondary structure model and structure-aware attention — trRosettaRNA2 accurately predicts RNA 3D structures and conformers by integrating pre-trained secondary structure priors and SS-aware attention.
• Introduction to RNNs — RNNs use a self-loop and hidden state to process sequential data, maintaining context over time.
• A new tool is revealing the invisible networks inside cancer — RNACOREX reveals hidden gene networks in cancer, predicting survival with AI-level accuracy and interpretable molecular explanations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine learning : nature.com subject feeds.