pipeline is really slow - consulting [D]

2026-05-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Software Development & Engineering · Depth: Advanced, medium

Summary

A robotics imitation learning pipeline, utilizing a frozen ResNet18 encoder and a ~50M parameter Diffusion Transformer (DiT) policy, is experiencing significantly slower training than expected. Despite running on an NVIDIA A4500 GPU with 48GB RAM and using bf16 mixed precision, GPU utilization remains low at 20-30%, while CPU utilization is consistently at ~100%. The pipeline processes 4 RGB 128x128x3 camera images and a 14-dimensional joint velocity vector, with data stored in Zarr. Throughput is only around 10 iterations/sec, causing an epoch of ~50k samples to take approximately 30 minutes. Neither increasing the batch size (currently 2) nor freezing the encoder substantially improved performance, and even synthetic data only yielded a ~50% throughput increase, indicating a persistent bottleneck.

Key takeaway

For Machine Learning Engineers debugging slow deep learning training pipelines, low GPU utilization coupled with high CPU usage strongly suggests a data loading or CPU-GPU synchronization bottleneck. You should systematically bisect your pipeline, starting with the network on fixed data, then adding the dataloader, and verifying performance with system-level profiling tools like `nsight systems` or `nvtop` to accurately identify and resolve the root cause.

Key insights

Low GPU utilization with high CPU points to a data loading or CPU-GPU synchronization bottleneck.

Principles

• Profiler outputs can be misleading; verify with system-level tools.
• Systematic pipeline bisection helps isolate performance bottlenecks.
• Optimal data chunking is crucial for efficient dataset access.

Method

Bisect the training pipeline by testing the network alone, then with backward/optimizer, then the dataloader alone, and finally combine components to pinpoint the performance bottleneck.

In practice

• Use system tools like `nvtop` or `nsight systems` for accurate GPU monitoring.
• Test network performance with fixed, in-memory batches to isolate dataloader issues.
• Optimize Zarr dataset chunking to be a multiple of the batch size.

Topics

• Robotics Imitation Learning
• Diffusion Transformers
• PyTorch Performance
• GPU Bottleneck
• Data Loading Optimization
• Zarr Dataset
• Profiling Tools

Best for: Machine Learning Engineer, Robotics Engineer, MLOps Engineer

Related on AIssential

• Your GPU Is Idle More Than You Think, and Your DataLoader Is the Reason — GPU idleness in PyTorch training often stems from data loading bottlenecks, fixable with DataLoader configuration.
• What Can We Do When Memory Becomes the New Bottleneck in Data Engineering? — Memory optimization for large datasets requires selecting the right tool based on project constraints.
• CPU Overload Despite Having iGPU: Here's Why? — Intel DL Streamer efficiently offloads video analytics tasks to iGPUs, freeing CPUs and optimizing real-time performance.
• Is Your Machine Learning Pipeline as Efficient as it Could Be? — Pipeline efficiency, not just model architecture, is the primary driver of machine learning team productivity and iteration speed.
• Part 12 -The 80GB Wall: GPU Infrastructure and Scheduling, Worked End to End — GPU memory, scheduling, and pre-training infrastructure decisions critically impact large model training success and cost.
• TraceLens: Democratizing AI Performance Analysis — TraceLens transforms complex AI profiler traces into actionable insights for performance optimization across various backends.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.