Multi-turn RL with Structural and Performance Aware Rewards for CUDA Kernel Generation

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

CudaPerf is a reflective Reinforcement Learning (RL) framework designed to generate optimized CUDA kernels by incorporating both verifiable execution rewards and structural code-aware rewards. Unlike existing RLVR approaches focusing solely on outcome-based signals, CudaPerf considers parallelization features such as memory coalescing, occupancy, Arithmetic Intensity, and synchronization patterns. The framework operates in two stages: an offline pairwise ranking module distinguishes strong and weak program candidates via contrastive comparisons, followed by an online RL training phase that jointly optimizes for correctness, performance, and structural efficiency using a unified reward signal. CudaPerf also utilizes iterative refinement with execution feedback. A new dataset was introduced, comprising 2.9k C to CUDA and 1k PyTorch to CUDA programs. Evaluated across C to CUDA and PyTorch to CUDA transformations, CudaPerf significantly outperforms baselines like Qwen-3-32B and CUDA Agent, achieving up to 5X and 3.32X improvements in speedup, and 17% and 7% improvements in correctness, respectively.

Key takeaway

For Machine Learning Engineers developing code generation models, CudaPerf demonstrates that integrating structural code properties into RL reward signals significantly boosts optimization. You should consider moving beyond purely outcome-based metrics like speedup and correctness to include parallelization features such as memory coalescing and occupancy. This approach can yield substantial performance gains, as shown by CudaPerf's 5X speedup improvement, making your generated CUDA kernels far more efficient.

Key insights

CudaPerf enhances LLM code generation by integrating structural and performance-aware rewards in a multi-turn RL framework.

Principles

• Combine execution and structural rewards for code optimization.
• Use contrastive learning for candidate ranking.
• Iterative refinement improves generated code.

Method

CudaPerf uses an offline pairwise ranking module for candidate distinction, then an online RL phase for joint optimization of correctness, performance, and structural efficiency via a unified reward signal and iterative refinement.

In practice

• Apply multi-turn RL to code generation.
• Incorporate parallelization features into reward signals.
• Develop datasets with diverse input configurations.

Topics

• Reinforcement Learning
• CUDA Kernel Generation
• Code Optimization
• Large Language Models
• Parallel Computing
• CudaPerf Framework

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Multi-turn RL with Structural and Performance Aware Rewards for CUDA Kernel Generation — CudaPerf enhances LLM code generation by integrating structural code properties with execution rewards in a multi-turn RL framework.
• daVinci-kernel: Co-Evolving Skill Selection, Summarization, and Utilization via RL for GPU Kernel Optimization — A reinforcement learning framework co-evolves skill selection, summarization, and utilization for GPU kernel optimization.
• Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design — Reward function design is paramount for multi-dimensional quality optimization in RL-based structured generation.
• How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF — ONNX Runtime and strategic batching are key to accelerating RLHF reward model inference, outperforming PyTorch on CPU.
• OptiML: An End-to-End Framework for Program Synthesis and CUDA Kernel Optimization — OptiML unifies LLM-driven code generation with hardware-aware search to optimize CUDA kernels for performance.
• Kernel Fusion in NVIDIA CUDA: Optimizing Memory Traffic and Launch Overhead — Kernel fusion combines GPU operations to reduce memory traffic and launch overhead for faster execution.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.