When Kubeflow meets Cilium: Debugging 60% idle GPUs in Kubernetes

2026-07-23 · Source: Cloud Native Computing Foundation · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, short

Summary

The article describes a problem where distributed training jobs on Kubernetes with Kubeflow and Cilium result in idle GPUs, despite all pods appearing healthy. This issue, observed by Ramkumar Nagaraj and Bingi Narasimha Karthik from Adobe, stems from a conflict between Kubernetes' topology-agnostic scheduling and Cilium's topology-aware network policies. Kubernetes places pods based on resource availability, while CiliumNetworkPolicy enforces zone boundaries for security or cost control. This can lead to training coordinators landing in one zone and GPU workers in another, with network policies silently blocking communication. This problem manifests as a hard block (training never starts), cross-zone latency (30-60% throughput drop), or cross-AZ egress costs. The fix involves adding Kubernetes-native YAML to the workload spec, specifically "nodeAffinity", "topologySpreadConstraints", and "toleration", to co-locate all communicating pods within a single network-open zone. This approach increased GPU utilization from 40% to 85% in lab tests.

Key takeaway

For MLOps Engineers deploying distributed ML training on Kubernetes, you must explicitly account for network topology in your workload specifications. If you rely solely on default Kubernetes scheduling, your GPU clusters risk silent performance degradation or complete job stalls due to topology-aware CNI policies like Cilium. Implement "nodeAffinity" and "topologySpreadConstraints" in your YAML to ensure co-location of communicating pods, preventing wasted GPU cycles and unexpected cross-zone egress costs. Proactively instrument GPU utilization and pod-zone metrics to detect these issues quickly.

Key insights

The core problem is a silent conflict between topology-agnostic schedulers and topology-aware network policies in distributed systems.

Principles

• Topology-aware CNIs can silently conflict with topology-agnostic schedulers.
• Individually correct system decisions can lead to collective failure.
• Silent network policies can cause idle resources or performance degradation.

Method

Apply Kubernetes-native YAML with "nodeAffinity", "topologySpreadConstraints", and "toleration" to co-locate distributed training pods within a single network zone.

In practice

• Use "nodeAffinity" to pin workloads to specific zones.
• Apply "topologySpreadConstraints" for pod co-location.
• Monitor GPU utilization and pod-zone metrics.

Topics

• Kubernetes Scheduling
• Kubeflow
• Cilium Network Policy
• Distributed ML Training
• GPU Optimization
• Topology Spread Constraints

Code references

• ram2valar/kubeflow-cilium-lab

Best for: MLOps Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Why your Kubernetes scheduler can't handle AI workloads — The default Kubernetes scheduler fails AI workloads due to partial-scheduling and lack of fabric topology awareness, necessitating specialized schedulers.
• Get Real-Time Visibility into GPU Usage Across Kubernetes Clusters — The GPU Usage Monitor provides integrated, real-time GPU observability for Kubernetes, preventing over-provisioning and scheduling delays.
• Finding zombies in our systems: A real-world story of CPU bottlenecks — Intermittent network issues on Ray ML jobs stemmed from CPU starvation caused by zombie memory cgroups from a crashing ECS agent.
• Validate Kubernetes for GPU Infrastructure with Layered, Reproducible Recipes — AI Cluster Runtime simplifies Kubernetes configuration for AI workloads via validated, version-locked, and reproducible recipes.
• The desktop infrastructure problem that kubernetes finally solves — Kubernetes can unify application and desktop infrastructure, offering operational consistency and enhanced security through containerized workspaces.
• Maximizing GPU Utilization: Heterogeneous Pipelines with Ray and Kubernetes — Ray optimizes heterogeneous, distributed AI workloads by managing diverse compute resources and handling failures across complex, multi-layered stacks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Cloud Native Computing Foundation.