Out-of-the-Box ROLL Support on AMD GPUs: Accelerating Reinforcement Learning at Scale

2026-06-01 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Intermediate, medium

Summary

Alibaba's open-source ROLL reinforcement learning framework now offers out-of-the-box support on AMD Instinct™ GPUs utilizing AMD ROCm™ software. Designed for large-scale, distributed RL, ROLL accelerates training for Large Language Models, powering reasoning and agentic behaviors. AMD collaborated with the ROLL development team, contributing upstream enhancements to ensure native compatibility without requiring code changes or custom builds. Key improvements include full vLLM sleep mode compatibility for versions ≥ 0.11.0, support for both vLLM Engine v0 and v1, and Ray compatibility fixes for versions ≥ 2.48 to correctly handle HIP_VISIBLE_DEVICES. ROLL features asynchronous execution for improved GPU utilization and agentic training for multi-agent workflows. Users are encouraged to leverage the official Docker image rlsys/roll_opensource for a seamless setup, which includes all necessary ROCm dependencies, Ray patches, vLLM engine support, and performance optimizations.

Key takeaway

For Machine Learning Engineers scaling reinforcement learning workloads for Large Language Models, you can now seamlessly deploy Alibaba's ROLL framework on AMD Instinct GPUs. This out-of-the-box compatibility, enabled by AMD's upstream contributions, eliminates complex setup and custom patching. You should leverage the official rlsys/roll_opensource Docker image and adjust gpu_memory_utilization in your training configurations to optimize performance and prevent potential CPU out-of-memory issues, ensuring efficient distributed training.

Key insights

Alibaba's ROLL framework now natively supports AMD Instinct GPUs via ROCm, accelerating large-scale reinforcement learning for LLMs.

Principles

• Distributed RL frameworks enhance LLM capabilities.
• Asynchronous execution improves GPU utilization.
• Upstream contributions ensure OOTB hardware compatibility.

Method

Users can run ROLL on AMD GPUs by building or pulling the official Docker image, preparing data, configuring training parameters like gpu_memory_utilization, and setting environment variables for single or multi-node execution.

In practice

• Utilize rlsys/roll_opensource Docker image for setup.
• Reduce gpu_memory_utilization to mitigate CPU OOM.
• Ensure Ray version ≥ 2.48 for correct GPU binding.

Topics

• Reinforcement Learning
• Large Language Models
• AMD ROCm
• ROLL Framework
• Distributed Training
• vLLM

Code references

• alibaba/ROLL
• ROCm/aiter

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• Reinforcement Learning from Human Feedback on AMD GPUs with verl and ROCm 7.0.0 — verl 0.6.0 with ROCm 7.0.0 enables efficient, scalable RLHF training on AMD Instinct MI300X GPUs.
• Libra: Efficient Resource Management for Agentic RL Post-Training — Libra efficiently manages resources for agentic RL post-training by dynamically optimizing GPU allocation and scheduling complex workloads.
• Deal between AMD and Meta could bring greater innovation to GenAI market — Strategic "chips-for-stock" partnerships are diversifying AI compute supply and challenging single-vendor dominance.
• Ollama now supports AMD graphics cards — Ollama now supports AMD GPUs on Windows and Linux, expanding local LLM accessibility.
• Taking on CUDA With ROCm: ‘One Step After Another’ — AMD's ROCm software stack is rapidly maturing, leveraging open-source tools and community engagement to challenge Nvidia's CUDA dominance.
• Streamlining Recommendation Model Training on AMD Instinct™ GPUs — ROCm training docker simplifies recommendation model training on AMD Instinct GPUs by optimizing sparse embedding handling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.