Scaling AI Inference Across Multiple GPUs Using NVIDIA TensorRT with Multi-Device Inference Support

2026-06-25 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Advanced, long

Summary

NVIDIA TensorRT 11.0 introduces multi-device inference support, enabling generative AI workloads to scale beyond single-GPU memory and compute limitations. This new feature integrates with NVIDIA NCCL, leveraging its high-performance collective operations like AllReduce and AllGather for distributed inference. The article details two primary parallelism strategies: tensor parallelism, which partitions model weights across GPUs, and context parallelism, which partitions input sequences. For context parallelism, three specific implementations are discussed: AllGather KV, Ring Attention, and DeepSpeed Ulysses. Benchmarks conducted on a single node with 8 GPUs using NVIDIA Cosmos 3 for video generation and FLUX.1 for image generation demonstrated that the Ulysses strategy consistently delivered the lowest end-to-end latency, especially for models with excessively long context lengths, while Ring Attention also showed good scaling for FLUX.1. This allows developers to deploy massive PyTorch models converted via Torch-TensorRT into C++ production environments.

Key takeaway

For AI Engineers scaling generative AI models that exceed single-GPU capacity, NVIDIA TensorRT 11.0's multi-device inference support provides a critical solution. You should integrate this feature, leveraging its native NCCL-powered distributed collective operations and exploring context parallelism strategies like DeepSpeed Ulysses. This enables efficient deployment of large PyTorch models, converted via Torch-TensorRT, into C++ production environments, significantly reducing latency for long-sequence workloads.

Key insights

TensorRT 11.0 scales generative AI inference across multiple GPUs using NCCL and parallelism strategies.

Principles

• Generative AI models frequently exceed single-GPU memory and compute.
• Distributed inference employs strategies like tensor and context parallelism.
• NVIDIA NCCL optimizes multi-GPU collective communication for inference.

Method

Create a TensorRT network with "IDistCollectiveLayer" for collectives, build the engine, set the NCCL communicator, and enqueue inference, running with OpenMPI.

In practice

• Download TensorRT 11.0 and install OpenMPI for multi-GPU setups.
• Convert PyTorch models using Torch-TensorRT for optimized deployment.
• Evaluate Ulysses for long-context generative media pipelines.

Topics

• NVIDIA TensorRT
• Multi-GPU Inference
• Context Parallelism
• Generative AI
• NVIDIA NCCL
• DeepSpeed Ulysses

Code references

• nvidia/cosmos
• NVIDIA/TensorRT

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Speed Up Unreal Engine NNE Inference with NVIDIA TensorRT for RTX Runtime — NVIDIA TensorRT for RTX significantly boosts neural network inference performance within Unreal Engine 5 on RTX GPUs.
• NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI — NVIDIA's latest platforms advance AI, robotics, and autonomous vehicles with significant performance and efficiency gains.
• Scaling Agentic Inference Across Heterogeneous Compute [Zain Asgar] - 757 — Heterogeneous inference orchestration across diverse hardware significantly improves AI workload efficiency and cost-per-token.
• Unlocking High-Performance Inference for DeepSeek with NVFP4 on NVIDIA Blackwell — Blackwell GPUs with NVFP4 quantization and TensorRT LLM dramatically boost MoE model inference performance.
• Maximizing GPU Utilization with NVIDIA Run:ai and NVIDIA NIM — Intelligent orchestration with NVIDIA NIM and Run:ai significantly boosts GPU utilization and inference performance for LLMs.
• Startup Boosts Scale-Up to 1000+ GPUs in a Single Domain — Disaggregated server design and software enable resilient, large-scale GPU inference clusters with flexible topologies.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.