REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning

2026-07-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

REGEN, a novel method published on 2026-07-21, addresses the high computational cost and infrastructure challenges of large-scale online reinforcement learning (RL) for eliciting advanced abilities in large language models (LLMs). While multi-teacher on-policy distillation (MOPD) helps decouple the RL stage, it still requires coupled inference and backward passes, limiting scalability. REGEN proposes "Replay-recycling for Expert-to-Generalist Distillation with Offline RL," which trains a generalist LLM by reusing replay memory—a free by-product of teachers' specialized RL training—and employing offline RL algorithms. This approach completely decouples rollout sampling from the backward training process, significantly reducing training costs. REGEN matches MOPD's accuracy across mathematical reasoning, code generation, and instruction following tasks at substantially lower expense.

Key takeaway

For Machine Learning Engineers scaling LLM capabilities across diverse tasks, REGEN offers a compelling alternative to traditional online RL or MOPD. If you are facing high computational costs or infrastructure limitations in post-training, you should explore REGEN's replay-recycling and offline RL approach. This method significantly reduces training expenses while maintaining accuracy in mathematical reasoning, code generation, and instruction following, potentially turning online RL into a more efficient data synthesis process for your projects.

Key insights

REGEN distills LLM knowledge using recycled replay memory and offline RL, decoupling training for cost efficiency.

Principles

• Online RL can serve as a data synthesis process.
• Decoupling rollout sampling from training reduces cost.
• Reusing replay memory enhances computational efficiency.

Method

REGEN trains a generalist LLM by recycling replay memory from specialized RL teachers and applying offline RL algorithms, completely decoupling rollout sampling from the backward training process.

In practice

• Distill LLM knowledge across diverse domains.
• Reduce computational cost of RL training.
• Enable large-scale post-training without heavy load.

Topics

• Reinforcement Learning
• Offline RL
• Large Language Models
• Knowledge Distillation
• Computational Efficiency
• Replay Memory

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning — Off-policy replay efficiently targets hard cases in LLM reinforcement unlearning, improving performance with minimal training overhead.
• Reinforcement Learning from Rich Feedback with Distributional DAgger — DistIL uses a forward cross-entropy objective with rich feedback to guarantee monotonic policy improvement in reinforcement learning.
• Efficient Exploration, Reasoning, and Training-Free MTP — Efficiency gains in LLMs are achievable through online RLHF, balanced reasoning control, and embedding-space multi-token prediction.
• Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution — ReMe dynamically manages LLM agent procedural memory, enabling smaller models to surpass larger, memoryless counterparts through experience-driven evolution.
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling — REVES enhances LLM multi-step reasoning by iteratively training on "near-miss" corrections and verifications, outperforming standard RL.
• Diffusion-GR2: Diffusion Generative Reasoning Re-ranker — Diffusion-GR2 enables fast, accurate generative reasoning re-ranking by converting autoregressive models to block-diffusion via targeted fine-tuning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.