Kernel Fusion in NVIDIA CUDA: Optimizing Memory Traffic and Launch Overhead

2026-07-10 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Advanced, medium

Summary

Kernel fusion in NVIDIA CUDA significantly optimizes GPU code by combining multiple operations into a single device kernel, thereby reducing intermediate memory traffic and kernel launch overhead. This technique addresses the common bottleneck where fast GPU compute is underutilized due to high-bandwidth device memory not being fully engaged. The article details three approaches: manual CUDA C++ kernel writing, implicit fusion using `torch.compile` (which leverages Torch Inductor and Triton), and explicit fusion via `cuda.compute` with iterators. Benchmarks on an NVIDIA GeForce RTX 4090 demonstrate that fused methods achieve approximately 850 GiB/s memory bandwidth, representing a 3x speedup over naive implementations by reducing global memory transfers from 3 GiB to 1 GiB. `cuda.compute` offers deterministic, portable, and composable solutions, built on the CUB library.

Key takeaway

For AI/ML engineers optimizing GPU workloads, understanding kernel fusion is critical to overcome memory bandwidth limitations. If your PyTorch models are bottlenecked by sequential operations, consider `torch.compile` for implicit fusion. For more predictable and fine-grained control, especially in library development or performance-critical applications, explore `cuda.compute` with its iterator-based explicit fusion, which offers CUB-backed optimizations and deterministic behavior without the maintenance burden of hand-written kernels.

Key insights

Kernel fusion combines GPU operations to reduce memory traffic and launch overhead for faster execution.

Principles

• GPU compute often bottlenecks on memory bandwidth.
• Fusing kernels keeps intermediate results in registers.
• Explicit fusion offers predictability over compiler-driven methods.

Method

Combine multiple GPU operations into a single kernel, either manually, implicitly via compilers like Torch Inductor, or explicitly using iterator-based frameworks like `cuda.compute`.

In practice

• Use `torch.compile` for automatic kernel fusion in PyTorch.
• Employ `cuda.compute` with `TransformIterator` for explicit fusion.
• Leverage CUB primitives for manual CUDA C++ kernel optimization.

Topics

• Kernel Fusion
• NVIDIA CUDA
• GPU Optimization
• torch.compile
• cuda.compute
• CUB Library

Code references

• NVIDIA/cuda-samples

Best for: Machine Learning Engineer, AI Engineer, Software Engineer

Related on AIssential

• Topping the GPU MODE Kernel Leaderboard with NVIDIA cuda.compute — The `cuda.compute` library enables Python developers to achieve near "speed-of-light" GPU performance with CUB primitives.
• Profiling in PyTorch (Part 2): From nn.Linear to a Fused MLP — Effective PyTorch profiling reveals kernel fusion, CPU overheads, and memory traffic bottlenecks for performance optimization.
• Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization — LLMs can optimize GPU kernels by integrating with micro-profiling tools and iterative search, achieving significant speedups and energy efficiency.
• A Guide to Understanding GPUs and Maximizing GPU Utilization — GPU bottlenecks often stem from inefficient data pipelines, not compute, requiring optimized data transfer and processing.
• AI in Multiple GPUs: Understanding the Host and Device Paradigm — Understanding Host-Device interaction, asynchronous execution, and CUDA Streams is fundamental for optimizing GPU performance.
• From Naive to Near-Peak: Building High-Performance GEMM Kernels with Gluon — Achieving near-peak GPU kernel performance requires explicit hardware control and a profiling-driven, iterative optimization methodology.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.