Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

2026-07-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Single-rollout Asynchronous Optimization (SAO) is a new method designed to enhance the stability and effectiveness of asynchronous Reinforcement Learning (RL) for post-training large language models (LLMs). Traditional synchronous RL pipelines are inefficient for long-horizon agentic tasks, and existing asynchronous systems often overlook training stability. SAO addresses this by replacing the group-wise sampling found in frameworks like GRPO with single-rollout sampling, using one rollout per prompt to mitigate off-policy effects and improve generalization. It further incorporates practical value-model training designs and a strict double-side token-level clipping strategy for optimization stability. SAO demonstrates stable training for one thousand steps and consistently outperforms GRPO and its variants on benchmarks such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. This approach is particularly effective in simulated online learning environments and has been deployed in the agentic RL pipeline for the open GLM-5.2 model (750B-A40B).

Key takeaway

For Machine Learning Engineers developing agentic LLMs, SAO offers a more stable and efficient asynchronous RL training paradigm. If you are struggling with off-policy effects or training instability in long-horizon tasks, consider implementing single-rollout sampling and token-level clipping. This approach, proven with GLM-5.2 (750B-A40B), can significantly improve performance on coding and reasoning benchmarks, especially in dynamic online learning settings.

Key insights

SAO improves asynchronous RL for LLMs by using single-rollout sampling and token-level clipping for stability and effectiveness.

Principles

• Single-rollout sampling reduces off-policy effects.
• Token-level clipping enhances optimization stability.
• Asynchronous RL improves efficiency for agentic tasks.

Method

SAO replaces group-wise sampling with single-rollout sampling (one rollout per prompt), integrates value-model training, and applies strict double-side token-level clipping for stable asynchronous RL.

In practice

• Deploy SAO for agentic RL in LLM training.
• Use single-rollout sampling for evolving environments.
• Apply token-level clipping for stable optimization.

Topics

• Reinforcement Learning
• Large Language Models
• Asynchronous Optimization
• Agentic AI
• Single-Rollout Sampling
• GLM-5.2

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• A Review of GLM-5: 744B-Parameter LLM Built for 200K Context and Agentic Training — GLM-5 bridges prompt-driven coding to agentic engineering via systems-level LLM advancements.
• Rethinking Groups in Critic-Free RLVR — The core function of groups in critic-free RL is to prevent false penalties, enabling stable single-rollout training via negative token filtering.
• AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention — AsyncTLS uses two-level sparse attention and asynchronous KV cache offloading for efficient long-context LLM inference.
• Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models — Dynamic sparsity schedules based on per-token actor-policy mismatch stabilize long-context RL, enabling significant speedups.
• Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding — Speculative decoding offers lossless acceleration for RL post-training rollouts, preserving model output distribution.
• Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning — The CoD framework trains LLMs via RL to achieve continuous learning and cross-domain generalization for long-lifecycle agents.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.