From Naive to Near-Peak: Building High-Performance GEMM Kernels with Gluon

2026-05-22 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, long

Summary

AMD's Gluon GEMM kernel optimization tutorial details the process of building high-performance General Matrix Multiply (GEMM) kernels for AMD MI350/MI355 (gfx950, CDNA4) GPUs. The tutorial demonstrates a journey from a 520 TFLOPS naive FP16 baseline to a 1489 TFLOPS near-peak kernel, achieving 99% MFMA efficiency through ten versions. This methodology, which emphasizes explicit control over hardware aspects via Gluon, also extends to BF8 and MXFP4 low-precision AI workloads, yielding 3257 TFLOPS (99.72% MFMA efficiency) and 5255 TFLOPS (92.41% MFMA efficiency) respectively. Key optimization steps include using AMD buffer operations, asynchronous copies, eliminating LDS bank conflicts, implementing software pipelining with "llirSched", managing register pressure through slicing, and applying XCD-aware workgroup remapping. The MXFP4 kernel specifically introduces a "scale pipeline" pattern for quantized data.

Key takeaway

For ML compiler engineers and kernel developers optimizing AI inference on AMD MI350/MI355 GPUs, you should adopt Gluon's explicit, profiling-driven methodology. This approach allows you to systematically identify and resolve bottlenecks, from LDS bank conflicts to register pressure and L2 locality. By applying techniques like software pipelining and register slicing, you can achieve near-peak MFMA efficiency and significantly boost TFLOPS for your compute-bound kernels.

Key insights

Achieving near-peak GPU kernel performance requires explicit hardware control and a profiling-driven, iterative optimization methodology.

Principles

• Profiling with metrics like MFMA efficiency reveals true bottlenecks.
• Explicitly managing layouts, pipelines, and register budgets optimizes GPU kernels.
• Systematic diagnosis is crucial when local fixes cause deeper regressions.

Method

Optimize kernels iteratively by identifying bottlenecks using detailed profiling (MFMA efficiency, VGPRs, spills), then applying techniques like software pipelining, register slicing, and architecture-specific remapping.

In practice

• Utilize Gluon for block-level GPU kernel programming.
• Implement multi-stage software pipelines to overlap operations.
• Slice output tiles (e.g., M- and N-halves) to reduce register pressure.

Topics

• GEMM Kernels
• Gluon Programming Model
• AMD MI355 Optimization
• MFMA Efficiency
• Low-Precision AI
• GPU Kernel Profiling

Code references

• ROCm/gfx950-gluon-tutorials
• triton-lang/triton
• triton-lang/triton
• ROCm/gfx950-gluon-tutorials

Best for: Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization — LLMs can optimize GPU kernels by integrating with micro-profiling tools and iterative search, achieving significant speedups and energy efficiency.
• Accelerating LLM Inference on AMD GPUs with Low-Latency GEMMs — LDS-Pipelined Split-K GEMM significantly accelerates LLM decode inference on AMD GPUs by optimizing small-M, large-K operations.
• A Guide to Understanding GPUs and Maximizing GPU Utilization — GPU bottlenecks often stem from inefficient data pipelines, not compute, requiring optimized data transfer and processing.
• Rigel: Reverse-Engineering the Metal 4.1 Tensor Compute Path on the Apple M4 Max GPU — Metal 4.1's fp8 matmul2d on Apple M4 Max is emulated, prioritizing memory footprint over raw performance.
• Performance Profiling on AMD GPUs - Part 4: Fortran OpenMP Offload Edition — Profiling Fortran OpenMP offload on AMD GPUs requires a systematic workflow to identify memory-bound kernels and optimize HSA_XNACK and workgroup sizes.
• Lambda's MLPerf Inference v6.0: hardware leap, software maturity, research breakthrough — Hardware, software, and routing optimizations significantly boost AI inference performance and reduce latency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.