Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization

2026-06-24 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

KernelPro is a closed-loop multi-agent system designed to automatically generate, profile, and iteratively optimize GPU kernel code by integrating large language model (LLM) code generation with hardware profiler feedback and pluggable bottleneck detection tools. The system introduces a semantic feedback operator that translates raw hardware metrics into actionable natural language guidance using expert heuristics, alongside a two-stage tool invocation architecture that classifies bottlenecks to orchestrate specialized profiling tools like ncu, SASS, and nsys. KernelPro also features a domain-adapted Monte Carlo Tree Search (MCTS) for cross-iteration learning and direct CuTe source-level code generation. On KernelBench, it achieved geometric mean speedups of 2.42x, 4.69x, and 5.30x on Levels 1, 2, and 3, respectively, setting new state-of-the-art performance. It also delivered a 1.23x speedup over hand-tuned Triton on VeOmni's expert-optimized MoE training kernels and demonstrated an 11.6% measured energy reduction at matched speed, a first for CUDA kernel coding agents.

Key takeaway

For Machine Learning Engineers optimizing GPU kernel performance, this research indicates that LLM-driven systems can significantly outperform hand-tuned code and achieve energy efficiency. You should consider integrating automated micro-profiling tools with LLM-based iterative refinement into your kernel development pipeline to achieve substantial speedups and reduce energy consumption, especially for complex architectures like Hopper WGMMA kernels. This approach offers a path to state-of-the-art optimization beyond manual efforts.

Key insights

LLMs can optimize GPU kernels by integrating with micro-profiling tools and iterative search, achieving significant speedups and energy efficiency.

Principles

• Expert heuristics can guide LLM optimization.
• Multi-stage profiling improves bottleneck detection.
• Iterative search refines LLM-generated code.

Method

KernelPro uses a closed-loop multi-agent system, combining LLM code generation with hardware profiler feedback, semantic feedback, and a domain-adapted MCTS for iterative GPU kernel optimization.

In practice

• Integrate profiler feedback into LLM workflows.
• Use roofline models for bottleneck classification.
• Explore MCTS for code optimization search.

Topics

• CUDA Kernel Optimization
• Large Language Models
• GPU Profiling
• Monte Carlo Tree Search
• Energy Efficiency
• Code Generation

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• OptiML: An End-to-End Framework for Program Synthesis and CUDA Kernel Optimization — OptiML unifies LLM-driven code generation with hardware-aware search to optimize CUDA kernels for performance.
• ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet) — Frontier LLMs struggle with multi-GPU kernel generation, but can occasionally produce novel, high-performance solutions.
• From Naive to Near-Peak: Building High-Performance GEMM Kernels with Gluon — Achieving near-peak GPU kernel performance requires explicit hardware control and a profiling-driven, iterative optimization methodology.
• I Built a C++ Backend So My GPU Would Stop Eating Air — Optimizing LLM inference requires VRAM-aware bin packing and efficient host-to-device data transfer to eliminate padding overhead.
• Sparser, Faster, Lighter Transformer Language Models — Reshaping LLM sparsity to fit GPU architecture significantly improves inference and training speed.
• Maximizing GPU Utilization with NVIDIA Run:ai and NVIDIA NIM — Intelligent orchestration with NVIDIA NIM and Run:ai significantly boosts GPU utilization and inference performance for LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.