Why Two GPUs Aren’t Twice as Fast

2026-07-19 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Intermediate, long

Summary

This article explains why using two GPUs for model training does not double the speed, even when the model fits on a single card. The bottleneck is the interconnect, the "wire" between GPUs, which becomes critical during the "all-reduce" operation where partial gradients are combined and broadcast. The author details different interconnect paths, from slowest (cross-socket PCIe, ~22 GB/s measured) to fastest (NVLink, a datacenter feature). An experiment using two consumer RTX 5090 GPUs on vast.ai measured the PCIe bus bandwidth at approximately 22 GB/s, significantly lower than the theoretical ~50 GB/s for a clean PCIe 5.0x16 link due to cross-socket communication (SYS topology). Rerunning the test with the suboptimal gloo backend, which routes through the CPU, dropped bandwidth to about 1.7 GB/s, a 13x reduction, proving the interconnect is the bottleneck. The article also provides a formula, T ≈ 2(N−1)/N · bytes / busbw, to predict all-reduce time, showing that for a 200 MB gradient, a 2-GPU setup takes ~9 ms, while an 8-GPU setup takes ~16 ms.

Key takeaway

For AI Engineers optimizing multi-GPU training, understand that interconnect bandwidth, not raw GPU power, often limits performance. Your system's GPU topology, visible via "nvidia-smi topo -m", directly impacts all-reduce speeds. Prioritize minimizing data transfer or overlapping communication with computation to mitigate this bottleneck. Do not assume linear scaling; instead, measure your specific interconnect performance to accurately predict training times and optimize resource allocation.

Key insights

Distributed GPU training speed is often bottlenecked by interconnect bandwidth, not GPU compute power.

Principles

• Interconnect speed dictates all-reduce performance.
• Topology (e.g., cross-socket PCIe) significantly impacts bandwidth.
• All-reduce cost scales with 2(N−1)/N factor.

Method

Measure interconnect bus bandwidth by sweeping all-reduce operations across message sizes using NCCL, then validate by comparing with a slower backend like gloo, and predict using the T ≈ 2(N−1)/N · bytes / busbw formula.

In practice

• Use "nvidia-smi topo -m" to identify GPU interconnect type.
• Time GPU operations with CUDA events for accuracy.
• Overlap communication with computation to hide latency.

Topics

• GPU Interconnects
• Distributed Training
• All-reduce
• NCCL
• PCIe
• NVLink
• Bandwidth Measurement

Code references

• NVIDIA/nccl-tests
• jtumale/ai-infra-lab

Best for: Machine Learning Engineer, AI Engineer, AI Student

Related on AIssential

• GPU Scaling from 1 to 261120 threads Part 2 — GPU performance scaling plateaus due to L2 bandwidth saturation and memory-bound operations, despite latency hiding.
• AI in Multiple GPUs: Understanding the Host and Device Paradigm — Understanding Host-Device interaction, asynchronous execution, and CUDA Streams is fundamental for optimizing GPU performance.
• Why Your Newest GPUs Might Be Slower Than Your Old Ones — A Real Latency Mystery From the NVIDIA… — Newer GPU generations may not offer universal performance improvements, particularly for small-message latency.
• AI’s Biggest Bottleneck Isn’t GPUs Anymore — AI's core bottleneck has moved from GPUs to High Bandwidth Memory, reshaping the industry's power dynamics.
• Accelerating Data Processing with NVIDIA Multi-Instance GPU and NUMA Node Localization — Data localization via NVIDIA MIG can significantly boost GPU performance under power constraints.
• A Guide to Understanding GPUs and Maximizing GPU Utilization — GPU bottlenecks often stem from inefficient data pipelines, not compute, requiring optimized data transfer and processing.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.