Attention Decode on AMD MI450 GPUs: A Gluon Kernel Optimization Guide

2026-07-27 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

AMD's blog details optimizing attention decode kernels for its MI450 series GPUs using Gluon, a Triton-based DSL. The MI450 introduces hardware features like the Tensor Data Mover (TDM) for asynchronous memory transfers, increased Local Data Share (LDS) to 320 KB, and workgroup clusters. The optimization guide focuses on four key areas: optimizing tensor layouts for WMMA operations, enhancing data loading via TDM by reshaping K/V tensors for direct-to-LDS transfers, implementing multi-stage pipelines (up to four-stage with triple buffering) to overlap memory access and computation, and parallelizing with Split-k using workgroup clusters to improve GPU saturation. An optimized Gluon kernel achieved 17.10 TB/s (85% of 20 TB/s peak) for GQA and 16.65 TB/s (83%) for MQA, demonstrating significant performance gains for memory-bound LLM inference workloads with context lengths up to 65536.

Key takeaway

For Machine Learning Engineers optimizing LLM inference on AMD MI450 GPUs, you should prioritize kernel-level optimizations that exploit MI450's unique hardware. Focus on custom Gluon kernels to manage tensor layouts, utilize TDM for direct memory-to-LDS transfers, implement multi-stage pipelining with triple buffering, and employ workgroup clusters for Split-k parallelism. This approach can yield significant performance gains, reaching up to 85% of peak HBM bandwidth, crucial for scaling agentic AI applications with long context lengths.

Key insights

Optimizing LLM attention decode on AMD MI450 GPUs requires deep hardware-software co-design, utilizing TDM, pipelining, and Split-k for peak HBM bandwidth.

Principles

• Memory-bound workloads benefit from direct-to-LDS data paths.
• Overlap memory transfers and compute to hide latency.
• Explicit tensor layout control is crucial for WMMA performance.

Method

Design high-performance kernels by optimizing WMMA tensor layouts, using TDM for direct global-to-LDS data loading, implementing multi-stage pipelines for latency hiding, and parallelizing with Split-k via workgroup clusters.

In practice

• Reshape K/V tensors to ensure TDM innermost dimension is ≥256 bytes.
• Use triple buffering in pipelines for longer lookahead.
• Employ multi-CTA programming in Gluon for Split-k parallelism.

Topics

• AMD MI450 GPUs
• Attention Decode
• Gluon Kernel Optimization
• LLM Inference Performance
• HBM Bandwidth
• Split-k Parallelization

Code references

• triton-lang/triton
• UoB-HPC/BabelStream

Best for: Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• QuickReduce FP4 Quantization and Benchmarking on MI355 — QuickReduce with FP4/INT4 quantization on MI355 significantly accelerates large-message all-reduce operations for LLM inference.
• From Tokens to Regions: CUDA-Sensitive Instruction Tuning for GPU Kernel Generation — CUDA sensitivity in LLM-generated kernels manifests at token and region levels, informing a low-cost instruction tuning method.
• Modern GPU Matmul Optimization — Modern GPU matmul optimization requires deep hardware understanding to surpass library performance.
• Building a monokernel for LLM inference on AMD MI300X - up to 3,300 output tokens/s per request [P] — Optimizing LLM inference on AMD MI300X by leveraging hardware die topology with a single GPU-resident monokernel.
• From Naive to Near-Peak: Building High-Performance GEMM Kernels with Gluon — Achieving near-peak GPU kernel performance requires explicit hardware control and a profiling-driven, iterative optimization methodology.
• Prefill/Decode Disaggregation: Why Your GPU Can’t Do Two Things at Once — LLM inference's prefill and decode phases have conflicting hardware demands, necessitating disaggregation for optimal performance and user experience.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.