Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism

2026-07-06 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Expert, medium

Summary

Nonuniform Tensor Parallelism (NTP) is an experimental framework designed to enhance Goodput in large-scale LLM training by elastically adapting to GPU unavailability. Building on existing methods like data replica dropping and fast checkpoint-restarts, NTP sustains high Goodput by dynamically adjusting the tensor parallelism degree and intelligently overlapping data resharding, thereby minimizing lost time and computational effort. Large-scale training, particularly with tensor parallelism across high-speed interconnects like NVIDIA NVLink (connecting up to 72 GPUs at 1,800 GB/s on NVIDIA Blackwell Ultra systems), is vulnerable to single GPU issues. NTP addresses this by reconfiguring TP groups (e.g., 8 GPUs to 7), power-boosting remaining GPUs to compensate for performance loss, and efficiently resharding during backward computation and parameter synchronization, often with less than 1% overhead. This approach represents a co-design of hardware and software for resilient AI training.

Key takeaway

For AI Architects designing large-scale LLM training infrastructure, consider integrating Nonuniform Tensor Parallelism (NTP) to enhance system resilience. Your designs should account for dynamic GPU availability by enabling adaptive tensor parallelism and power-boosting capabilities. This approach minimizes Goodput loss during transient hardware interruptions, ensuring your training jobs maintain efficiency and progress even when individual devices fluctuate. Explore NVIDIA Megatron Core's NTP implementation for practical application.

Key insights

NTP dynamically adapts tensor parallelism and power to maintain LLM training Goodput despite GPU failures, minimizing overhead.

Principles

• Elastic adaptation improves Goodput in large-scale LLM training.
• Co-designing hardware and software enhances resilience.
• Overlapping resharding minimizes performance overhead.

Method

NTP dynamically reconfigures tensor parallelism degree, power-boosts active GPUs in affected groups, and overlaps data resharding with backward computation to sustain Goodput during GPU interruptions.

In practice

• Implement dynamic TP degree adjustment.
• Utilize power-boosting for performance compensation.
• Overlap data resharding with backward passes.

Topics

• LLM Training
• Tensor Parallelism
• Goodput Optimization
• GPU Resilience
• NVIDIA Blackwell
• Megatron Core
• Data Resharding

Code references

• NVIDIA/nvidia-resiliency-ext
• NVIDIA/Megatron-LM

Best for: Research Scientist, MLOps Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron — NVIDIA enables large-scale LLM training with higher-order optimizers like Muon through specialized distributed computing techniques.
• Zyphra Introduces Tensor and Sequence Parallelism (TSP): A Hardware-Aware Training and Inference Strategy That Delivers 2.6x Throughput Over Matched TP+SP Baselines — TSP simultaneously shards model weights and token sequences, resolving GPU memory bottlenecks for long-context transformers.
• How NVIDIA Extreme Hardware-Software Co-Design Delivered a Large Inference Boost for Sarvam AI’s Sovereign Models — Full-stack co-optimization of model design, kernels, scheduling, and hardware significantly boosts LLM inference performance.
• Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization — Parallel Track Transformers reduce GPU synchronization for faster, more efficient LLM inference.
• The Most Expensive Thing in AI Training Is Waiting: Taming the Long Tail in RL — Reclaiming idle GPU time during RL training's "long tail" significantly boosts efficiency without sacrificing model quality.
• Sakana AI and NVIDIA Introduce TwELL with CUDA Kernels for 20.5% Inference and 21.9% Training Speedup in LLMs — TwELL and new CUDA kernels exploit LLM sparsity for significant speedups and energy savings on NVIDIA GPUs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.