ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information

2026-06-06 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

ASymPO (Asymmetric-Scale Policy Optimization) is a novel method addressing distribution drift in asynchronous reinforcement learning for large language model (LLM) post-training. This drift, caused by stale responses, creates a scale-imbalance failure mode where positive and negative loss terms appear at different negative log-probability scales, destabilizing training. Unlike standard behavior-corrected methods that demand complex infrastructure like token-aligned behavior log-probabilities and policy versioning, ASymPO normalizes each response's token loss by its current average token negative log-probability. This approach restores response-level zero-sum balance and maintains a nonzero learning signal using only current-policy probabilities. The paper also introduces Scaled Policy Optimization (SPO) as a fixed negative-scaling baseline. Evaluated on asynchronous mathematical reasoning post-training across Qwen3-1.7B-Base, Qwen3-4B-Base, and LLaMA-3.2-3B-Instruct models, ASymPO and SPO demonstrated stable training, unlike naive loss and GPG which collapsed. ASymPO significantly simplifies the rollout–learner interface by eliminating the need for behavior log-probability transport and policy-version metadata.

Key takeaway

For Machine Learning Engineers designing asynchronous LLM post-training pipelines, you should consider adopting ASymPO. This method stabilizes training by adaptively balancing loss contributions without requiring complex behavior-policy probabilities or policy-version bookkeeping. Implementing ASymPO simplifies your rollout–learner interface, reducing infrastructure overhead and avoiding the training collapses observed with naive current-policy objectives, thereby streamlining your development and deployment efforts.

Key insights

Asynchronous LLM post-training can be stabilized by adaptively scaling current-policy loss, removing behavior-policy dependencies.

Principles

• Asynchronous RL without behavior correction risks scale-imbalance.
• Zero-sum advantages alone do not guarantee balanced loss.
• Normalizing loss by current log-probability scale restores balance.

Method

ASymPO normalizes each response's token loss by its current average token negative log-probability, using a stop-gradient operator to balance loss contributions.

In practice

• Implement ASymPO to simplify rollout–learner interface.
• Eliminate behavior log-probability and policy version transport.
• Apply current-policy-only objectives for LLM post-training.

Topics

• ASymPO
• Asynchronous RL
• LLM Post-Training
• Policy Optimization
• Distribution Drift
• Mathematical Reasoning

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information — ASymPO stabilizes asynchronous LLM post-training by normalizing token loss with current policy probabilities, eliminating the need for behavior information.
• The Death of RLHF: A Practitioner’s Guide to the New Post-Training Stack — New post-training methods like GRPO, RLVR, and DAPO have replaced RLHF for reasoning tasks, improving scalability and efficiency.
• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training — Adaptive critic utilization in LLM post-training can reduce advantage variance more effectively than fixed baselines.
• DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training — Asynchronous RL for LLMs can achieve high efficiency and convergence by maintaining policy consistency and dynamic resource orchestration.
• Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries — Asynchronous RL training disaggregates inference and training to overcome generation bottlenecks and GPU idle time.
• RL Scaling Laws for LLMs — Scaling laws, while precise for LLM pretraining, are more complex and context-dependent in reinforcement learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.