How we keep GPUs reliable across Databricks AI

2026-07-01 · Source: Databricks · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Advanced, medium

Summary

Databricks AI details its approach to maintaining GPU reliability for massive-scale training workloads, addressing continuous failures across hardware, fabric, and software. The article identifies three primary failure categories: crashed jobs (often due to "NCCL watchdog timeout" messages), silent slowdowns (from degraded hardware like thermal throttling or interconnect issues, indicated by DCGM throttle reasons like "HW_SLOWDOWN" or "HW_THERMAL_SLOWDOWN"), and numerical corruption (memory faults or propagating errors). Given a conservative 1% annualized GPU failure rate, a 256-GPU job running for 30 days has a 19% chance of failure, rising to 57% for 1,024 GPUs, making failures expected. Databricks' solution involves stress testing with diverse, cutting-edge workloads and a multi-stage "gpu-monitor" health check system that operates across the node lifecycle.

Key takeaway

For MLOps Engineers building large-scale distributed GPU training infrastructure, you must anticipate and engineer for continuous hardware failures. Your reliability strategy should extend beyond basic crash detection to include identifying silent slowdowns and numerical corruption. Implement a multi-stage health check system, like Databricks' "gpu-monitor", to validate hardware pre-workload, monitor during execution, and verify inter-node fabric between jobs, ensuring robust and cost-effective model training.

Key insights

Distributed GPU training at scale necessitates proactive, multi-layered reliability engineering to counter expected hardware failures.

Principles

• GPU failure rates increase significantly with scale and duration.
• Diverse workloads expose unique platform vulnerabilities.
• Cumulative downtime, not just flap count, impacts training.

Method

Implement a multi-stage "gpu-monitor" service with active bootstrap checks (pre-workload), passive continuous checks (during workload), and periodic multi-node active checks (inter-workload) to cover the entire node lifecycle.

In practice

• Tune "NCCL_IB_TIMEOUT" for InfiniBand resilience.
• Monitor DCGM throttle reasons like "HW_SLOWDOWN".
• Validate NCCL collective bandwidth across payload sizes.

Topics

• GPU Reliability
• Distributed Training
• NCCL Watchdog Timeout
• Hardware Health Checks
• Databricks AI
• InfiniBand Performance

Best for: MLOps Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Fixing GPU Starvation in Large-Scale Distributed Training — GPU underutilization often stems from data I/O and format transformation bottlenecks, not model complexity.
• AMD Device Metrics Exporter v1.4.2: Enhanced Observability, Deeper RAS Insights, and Smarter GPU Telemetry for Modern HPC & AI Clusters — DME v1.4.2 enhances GPU observability with process-level context, structured error reporting, and detailed performance constraint metrics.
• Must-Know Failure Modes in Distributed Systems — Distributed systems' "up" state is deceptive; recurring failure patterns demand specific, known defenses beyond simple health checks.
• Introducing NVIDIA Fleet Intelligence for Real-Time GPU Fleet Visibility and Optimization — NVIDIA Fleet Intelligence provides comprehensive, real-time monitoring and integrity verification for large-scale GPU and CPU fleets.
• Startup Boosts Scale-Up to 1000+ GPUs in a Single Domain — Disaggregated server design and software enable resilient, large-scale GPU inference clusters with flexible topologies.
• [P] ML training cluster for university students — Building an ML training cluster for students requires balancing cost, scalability, and ease of use, often favoring dedicated hardware over cloud.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.