Accelerating Data Processing with NVIDIA Multi-Instance GPU and NUMA Node Localization

2026-02-19 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, medium

Summary

NVIDIA's Ampere, Hopper, and Blackwell data center GPUs feature non-uniform memory access (NUMA) behaviors despite exposing a single memory space, which can impact performance and power efficiency as bandwidth increases. This analysis explores the GPU memory hierarchy, detailing how data transfers over die-to-die links affect power and performance, particularly due to increased latency and power limitations on larger GPUs. It then reviews NVIDIA's Multi-Instance GPU (MIG) mode as a strategy to achieve data localization by partitioning a single GPU into multiple instances, thereby minimizing inter-NUMA node data transfers. Experimental results using the Wilson-Dslash stencil operator demonstrate that MIG mode can achieve speedups of up to 2.25x at lower GPU power limits (e.g., 400 W) compared to unlocalized approaches, though its benefits diminish or reverse at higher power limits due to communication overhead.

Key takeaway

For AI Engineers optimizing GPU workloads under strict power limits, implementing MIG-based NUMA node localization can yield substantial performance gains, potentially up to 2.25x faster at 400 W. However, you should carefully evaluate the trade-offs, as MIG's benefits decrease or become negative at higher power limits due to increased inter-process communication overhead. Consider this approach for smaller workloads that fit within MIG instances and have minimal cross-instance communication.

Key insights

Data localization via NVIDIA MIG can significantly boost GPU performance under power constraints.

Principles

• Minimize inter-NUMA node data transfers.
• Coherent L2 caching reduces data refetching.

Method

Partition a GPU into multiple MIG instances, assigning one instance per NUMA node to isolate memory access and eliminate L2 fabric transfers. Use `CUDA_VISIBLE_DEVICES` for process-to-instance mapping.

In practice

• Use MIG for power-constrained GPU workloads.
• Create one MIG instance per NUMA node.
• Map MPI processes to specific MIG instances.

Topics

• NVIDIA GPUs
• NUMA Architecture
• Multi-Instance GPU
• Data Locality
• Performance Optimization

Code references

• lattice/quda

Best for: AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• Maximize AI Infrastructure Throughput by Consolidating Underutilized GPU Workloads — GPU partitioning via MIG or time-slicing significantly improves resource utilization and throughput in Kubernetes AI deployments.
• NVIDIA drops DGX Station for Windows (1-Trillion Parameter desktop). Who else is ready to run LLaMA-Behemoth locally? — Aggressive quantization remains crucial for local inference of trillion-parameter models, even with powerful new hardware.
• GPU Scaling from 1 to 261120 threads Part 1 — GPU performance scales non-linearly with threads due to memory latency and inefficient warp utilization.
• Scaling AI Inference Across Multiple GPUs Using NVIDIA TensorRT with Multi-Device Inference Support — TensorRT 11.0 scales generative AI inference across multiple GPUs using NCCL and parallelism strategies.
• From Naive to Near-Peak: Building High-Performance GEMM Kernels with Gluon — Achieving near-peak GPU kernel performance requires explicit hardware control and a profiling-driven, iterative optimization methodology.
• Fixing GPU Starvation in Large-Scale Distributed Training — GPU underutilization often stems from data I/O and format transformation bottlenecks, not model complexity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.