I ported DeepMind's DiscoRL learning rule from JAX to PyTorch

2026-03-10 · Source: Machine Learning ML & Generative AI News · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, quick

Summary

A new PyTorch implementation of DeepMind's DiscoRL learning rule is now available, ported from its original JAX framework. This port, hosted on GitHub and with weights on Hugging Face, includes a Colab notebook for immediate experimentation and an API. The developer undertook this project to facilitate the application of DiscoRL to Large Language Model (LLM) training, a domain predominantly utilizing PyTorch. While some nuances regarding action space still require refinement, the PyTorch version aims to make this advanced reinforcement learning technique more accessible for broader research and development, particularly within the LLM community.

Key takeaway

For NLP engineers and AI scientists exploring advanced reinforcement learning for LLMs, you should investigate this new PyTorch port of DeepMind's DiscoRL. This port removes a significant framework barrier, allowing you to directly apply and experiment with DiscoRL within your existing PyTorch-based LLM training pipelines. Leverage the provided Colab notebook to quickly test its applicability to your specific models and tasks.

Key insights

DeepMind's DiscoRL learning rule is now available in PyTorch, enhancing accessibility for LLM training.

Method

The method involved porting the DiscoRL learning rule from its original JAX implementation to PyTorch, including creating a GitHub repository, a Colab notebook, and an API, with weights hosted on Hugging Face.

In practice

• Experiment with DiscoRL for LLM training.
• Utilize the provided Colab notebook.
• Integrate the DiscoRL API into PyTorch projects.

Topics

• DiscoRL
• Reinforcement Learning
• PyTorch
• JAX
• Large Language Models

Code references

• asystemoffields/disco-torch

Best for: NLP Engineer, AI Scientist, Research Scientist, AI Engineer, Machine Learning Engineer, Deep Learning Engineer

Related on AIssential

• From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space — Optimizing the marginal distribution P(y) in pre-train space enhances LLM reasoning and exploration beyond conditional P(y|x) optimization.
• Reinforcement Learning Course — This course provides a structured deep dive into Reinforcement Learning, from fundamentals to advanced applications.
• Reinforcement Learning from Human Feedback on AMD GPUs with verl and ROCm 7.0.0 — verl 0.6.0 with ROCm 7.0.0 enables efficient, scalable RLHF training on AMD Instinct MI300X GPUs.
• ICRL: Learning to Internalize Self-Critique with Reinforcement Learning — Jointly training a solver and critic with reinforcement learning enables LLMs to internalize self-critique and improve unassisted.
• ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learning — ReSum enhances LLM reasoning by integrating self-summarization to compress rollouts and reduce error propagation.
• A Practical Guide to PyTorch: The Deep Learning Framework Built for People Who Think in Code — PyTorch's dynamic computation graphs and Python integration empower flexible deep learning research and deployment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning ML & Generative AI News.