GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining

2026-07-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Expert, extended

Summary

GIFT is a novel geometry-informed gradient scaling method designed to address the communication bottleneck in large language model (LLM) pretraining. It improves low-precision gradient communication, specifically using FP8, by transforming anisotropic gradients into a near-isotropic space before quantization. This approach makes low-precision representations substantially more faithful to their high-precision counterparts, avoiding performance degradation seen in existing Euclidean quantization methods. GIFT does not alter the optimizer or training recipe, focusing solely on the coordinate system for communication. It employs a simplified K-FAC-based geometry-aware transformation with low-rank approximation and selective application to balance computational overhead and communication reduction. Empirical results with Llama-300M and Llama-600M models show GIFT reduces Llama-600M pretraining time by 7.6% on 64 NVIDIA GH200 Superchips, while also improving downstream task preservation compared to direct Euclidean FP8 communication. This method achieves a 75.0% reduction in gradient communication volume.

Key takeaway

For AI Architects designing large-scale LLM pretraining systems, you should consider geometry-informed gradient communication to mitigate performance degradation from low-precision formats. Implementing GIFT can reduce pretraining time by 7.6% for models like Llama-600M on 64 NVIDIA GH200 Superchips, while significantly improving downstream task preservation. This approach allows you to achieve communication efficiency without sacrificing model quality, especially as GPU counts and model sizes increase.

Key insights

Anisotropic gradients cause low-precision quantization errors; transforming them into isotropic coordinates improves fidelity.

Principles

• Quantize gradients in geometry-aware coordinates for better fidelity.
• Selective application of complex methods balances cost and benefit.
• Input-side geometry captures most fidelity benefits.

Method

GIFT transforms gradients using a low-rank K-FAC approximation, quantizes in the new coordinate system, communicates, and maps back. This is applied selectively to vulnerable layers.

In practice

• Profile layers for numerical vulnerability to guide optimization.
• Use rank-32 approximation for K-FAC factors.
• Apply geometry-aware transforms only to critical layers.

Topics

• LLM Pretraining
• Gradient Communication
• Low-Precision Training
• FP8 Quantization
• K-FAC Approximation
• NVIDIA GH200 Superchips

Best for: AI Engineer, NLP Engineer, Research Scientist, Machine Learning Engineer, AI Scientist, AI Architect

Related on AIssential

• GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining — Transforming gradients into a near-isotropic space before low-precision quantization significantly improves fidelity and performance.
• SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication — SCAPE enables aggressive LLM training sparsification by leveraging AdamS's first-moment for stability and efficiency.
• AI Model Co-Design: Hardware-Friendly LLM Design — Hardware-friendly LLM design optimizes throughput and interactivity by aligning model architecture with GPU capabilities.
• PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training — Polynomial preconditioning reshapes weight singular-value spectra to improve LLM training stability and efficiency without inference overhead.
• Scaling LLMs horizontally: hidden-state coupling without weight modification [R] — Residual Coupling enables horizontal scaling of frozen LLMs via linear bridges, enhancing performance without weight modification.
• Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism — Lossless compression for model-parallel training is achieved by constraining activations and gradients to low-rank subspaces.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.