Unlock Exascale Performance on NVIDIA GB200 NVL72 with Slurm Topology-Aware Job Scheduling

2026-05-21 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, long

Summary

NVIDIA's GB200 NVL72 system, featuring 72 Blackwell GPUs and 130 TB/s NVLink bandwidth, delivers exascale compute for AI and HPC workloads, enabling over 2.6x MLPerf training performance and real-time trillion-parameter model inference. To fully utilize this infrastructure in shared clusters, topology-aware job scheduling with Slurm is essential. The new topology/block plugin, introduced in Slurm 23.11, aligns jobs with GB200 NVL72's NVLink domains, supporting larger job segment sizes up to 18 nodes. Simulations on a 5,000-node (20,000 GPU) cluster over seven days demonstrated that this approach minimizes fragmentation by concentrating small jobs on specific nodes (N17, N18) and achieves 94.2% GPU occupancy, only about 1% below theoretical maximums, proving high utilization without performance penalties.

Key takeaway

For MLOps Engineers managing NVIDIA GB200 NVL72 clusters, optimizing workload placement is crucial for maximizing performance and utilization. You should implement Slurm's topology/block plugin (Slurm 23.11+) to align jobs with NVLink domains, configuring larger jobs (e.g., 32+ nodes) with 16-node segments and smaller jobs with 2-8 node segments. Continuously monitor fragmentation and GPU occupancy, using simulation tools to validate policy changes before production deployment to sustain high efficiency.

Key insights

Topology-aware scheduling with Slurm's topology/block plugin is crucial for maximizing NVIDIA GB200 NVL72 performance and utilization.

Principles

• Workload placement is critical for accelerated infrastructure performance.
• Schedulers must align jobs with network topology.
• Optimal segment size depends on workload characteristics.

Method

Implement Slurm's topology/block plugin for GB200 NVL72-aware job placement. Prioritize large jobs (≥64 GPUs) with maximum NVLink domain access and proportional segment sizing. Use simulation to validate new scheduling policies before production deployment.

In practice

• Deploy Slurm 23.11 with the topology/block plugin.
• Configure large jobs (≥32 nodes) with 16-node segments.
• Monitor fragmentation and adjust segment sizes dynamically.

Topics

• NVIDIA GB200 NVL72
• Slurm Scheduling
• Topology-aware Scheduling
• GPU Cluster Management
• AI Workload Optimization
• NVLink Fabric

Best for: AI Architect, MLOps Engineer, DevOps Engineer

Related on AIssential

• Achieving Peak System and Workload Efficiency on NVIDIA GB200 NVL72 with Slurm Block Scheduling — NVIDIA GB200 NVL72's rack-scale NVLink coherence requires Slurm's topology/block plugin for optimal scheduling.
• Running AI Workloads on Rack-Scale Supercomputers: From Hardware to Topology-Aware Scheduling — NVIDIA Mission Control and related tools bridge hardware topology with scheduler abstractions for optimal AI/HPC workload placement.
• Machine Learning Model Development on HPC Clusters with SLURM: From Basics to Production — HPC clusters with SLURM provide scalable resource management for production-grade machine learning model development.
• GPU Scaling from 1 to 261120 threads Part 1 — GPU performance scales non-linearly with threads due to memory latency and inefficient warp utilization.
• Nautilus: An Auto-Scheduling Tensor Compiler for Efficient Tiled GPU Kernels — Nautilus automates tensor kernel optimization from math specifications, outperforming existing compilers.
• Startup Boosts Scale-Up to 1000+ GPUs in a Single Domain — Disaggregated server design and software enable resilient, large-scale GPU inference clusters with flexible topologies.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.