ROCm with PyTorch and PyTorch Lightning seems to still suck for research [D]

2026-05-16 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, short

Summary

A user's experience with AMD's ROCm platform for machine learning research indicates significant issues, particularly with PyTorch and PyTorch Lightning. After procuring an RX 7900XTX, the user encountered widespread NaN errors during the backward pass when porting a flow matching model (SANA Architecture) codebase that ran flawlessly on NVIDIA RTX3090s. Attempts to resolve these issues by switching between bf16 and fp32 precisions or adjusting environment variables were unsuccessful. While a standard nanoGPT training script ran perfectly, the user's intuition suggests ROCm's stack is robust only for well-established codebases and fragile with slightly uncommon code. Comments from other users corroborate these findings, citing similar NaN issues, differing operator behavior compared to CUDA, and a lack of clear documentation on critical differences for ROCm users.

Key takeaway

For AI Engineers evaluating AMD GPUs for deep learning research, you should anticipate potential stability issues and NaN errors, especially with custom or less common PyTorch codebases. Be prepared for a non-"plug and play" experience, requiring significant debugging and potential workarounds like adjusting precision or compiler settings. Your existing CUDA-optimized code may not translate seamlessly, necessitating a thorough validation process for backward passes and custom operations.

Key insights

ROCm remains challenging for ML research due to instability and NaN issues with less common PyTorch codebases.

Principles

• ROCm's stability varies significantly with codebase commonality.
• CUDA and ROCm operator behaviors can differ critically.

In practice

• Expect NaN issues during backward passes on ROCm.
• Consider using Python 3.12 for ROCm compatibility.
• Explicitly set compiler to XLA for ROCm with Elixir/NX.

Topics

• ROCm
• PyTorch
• PyTorch Lightning
• NaN Issues
• GPU Programming

Best for: AI Engineer, NLP Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• Taking on CUDA With ROCm: ‘One Step After Another’ — AMD's ROCm software stack is rapidly maturing, leveraging open-source tools and community engagement to challenge Nvidia's CUDA dominance.
• PyTorch Offline Tuning with TunableOp — Offline tuning separates BLAS kernel optimization from model execution, improving flexibility and efficiency.
• What do you need to learn to be an AI Engineer in 2026? Where to Learn it? What to build with it? — AI engineering prioritizes building reliable, production-grade systems around models, requiring deep understanding across the entire ML lifecycle.
• Accelerating ComfyUI Workflows on AMD Instinct™ MI355X GPUs with ROCm — AMD Instinct MI355X GPUs significantly accelerate ComfyUI generative AI workflows over NVIDIA B200.
• A Guide to Understanding GPUs and Maximizing GPU Utilization — GPU bottlenecks often stem from inefficient data pipelines, not compute, requiring optimized data transfer and processing.
• Sparser, Faster, Lighter Transformer Language Models — Reshaping LLM sparsity to fit GPU architecture significantly improves inference and training speed.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.