Mixed-Precision Communication-Avoiding SGD for Generalized Linear Models on GPUs

2026-06-18 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Cloud Computing & IT Infrastructure · Depth: Expert, extended

Summary

Mixed-Precision Communication-Avoiding SGD (CA-SGD) for Generalized Linear Models on GPUs overcomes communication bottlenecks in distributed SGD by amortizing communication over s iterations. This method replaces s consecutive AllReduces with a single AllReduce of an sb×sb Gram matrix, leveraging modern GPUs' matrix hardware and reduced-precision formats like BF16 to accelerate Gram GEMM and shrink traffic. A finite-precision analysis decomposed local rounding error into nine independent precision choices, leading to "Recipe C". This recipe stores the input matrix and margin vector in low precision (BF16), computes the Gram matrix with high-precision accumulation (FP32), communicates it in high precision (FP32), and performs inner recurrence and weight updates in high precision (FP32). On NERSC Perlmutter A100 GPUs, "Recipe C" matched FP32 SGD loss within 0.5% on logistic, linear, and Poisson problems, achieving 5.1–6.8× speedup over FP32 SGD on epsilon, SUSY, HIGGS, synth, and Poisson-synth datasets. A variant, "Recipe D", reached up to 6.8× speedup but with a larger relative loss gap.

Key takeaway

For Machine Learning Engineers optimizing distributed Generalized Linear Models on NVIDIA A100 GPUs, adopting mixed-precision Communication-Avoiding SGD (CA-SGD) can yield substantial speedups. You should implement "Recipe C" by storing inputs in BF16 and using FP32 accumulation for critical operations like Gram GEMM. This approach matches FP32 SGD accuracy within 0.5% while achieving 5.2–6.2× speedup, significantly reducing training time. Consider "Recipe D" for higher speedups on large-n datasets, but verify its accuracy for your specific problem.

Key insights

Mixed-precision CA-SGD significantly accelerates distributed GLM training on GPUs by optimizing communication and computation.

Principles

• Communication, not computation, limits distributed SGD.
• Amortize communication over multiple iterations.
• Precision choices must align with error sensitivities.

Method

The proposed "Recipe C" for mixed-precision CA-SGD involves BF16 storage for inputs/margins, BF16-input/FP32-accumulate for Gram GEMM, and FP32 for inner correction, residuals, master weights, and Gram AllReduce.

In practice

• Use BF16 for input/margin storage on A100 GPUs.
• Employ FP32 accumulation for Gram GEMM with BF16 inputs.
• Keep inner correction and master weights in FP32.

Topics

• Communication-Avoiding SGD
• Mixed Precision Training
• Generalized Linear Models
• NVIDIA A100 GPUs
• Distributed Optimization
• Finite-Precision Analysis

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging — LOSCAR-SGD combines local training, sparse communication, and overlap with a delay-corrected merge for efficient distributed learning.
• Training a Model with Limited Memory using Mixed Precision and Gradient Checkpointing — Mixed precision and gradient checkpointing enable training large models on memory-limited hardware.
• On the Utility of Equal Batch Sizes for Inference in Stochastic Gradient Descent — Equal batch sizes can consistently estimate SGD asymptotic covariance with memory efficiency and bias correction.
• From Tokens to Regions: CUDA-Sensitive Instruction Tuning for GPU Kernel Generation — CUDA sensitivity in LLM-generated kernels manifests at token and region levels, informing a low-cost instruction tuning method.
• stochastic gradient descent #maths #statistics #datascience #machinelearning #dataanlysis — Faster iteration with noisy gradients often outperforms slow, precise convergence in machine learning.
• Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity — Rescaled ASGD uses worker-specific stepsize rescaling to correct objective bias in heterogeneous asynchronous distributed learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.