Unlocking asynchronicity in continuous batching

2026-05-14 · Source: Hugging Face - Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

Hugging Face has released an update on asynchronous batching for efficient Large Language Model (LLM) inference, building on their previous work on continuous batching. This technique addresses the inefficiency of synchronous batching, where the CPU and GPU alternate, leading to significant idle time for both. By disentangling CPU batch preparation from GPU computation, asynchronous batching allows both components to run in parallel. The article demonstrates that this approach can reduce total generation time for an 8B model generating 8K tokens with a batch size of 32 from 300.6 seconds to 234.5 seconds, achieving a 22% speedup and increasing GPU utilization from 76.0% to 99.4%. This performance gain is achieved without new kernels or model changes, relying instead on careful coordination of hardware using CUDA streams and events.

Key takeaway

For MLOps engineers optimizing LLM inference, adopting asynchronous batching is crucial to maximize GPU utilization and reduce generation latency. By implementing CUDA streams and events, you can eliminate idle gaps between CPU and GPU operations, achieving significant speedups without modifying model kernels. Consider integrating this approach into your continuous batching pipelines to improve throughput for long generation tasks, especially with high-cost GPUs like the H200.

Key insights

Asynchronous batching separates CPU and GPU workloads to maximize parallel execution and GPU utilization during LLM inference.

Principles

• Operations in different CUDA streams can run concurrently.
• CUDA events enforce synchronization across streams.
• Alternating input/output slots prevents race conditions.

Method

Utilize CUDA streams for H2D transfer, compute, and D2H transfer, synchronizing them with CUDA events. Employ alternating memory slots and a memory pool for CUDA graphs to prevent race conditions and manage carry-over tokens.

In practice

• Instrument continuous batching code to profile CPU/GPU activity.
• Use non-default CUDA streams for concurrent GPU operations.
• Implement CUDA events for inter-stream synchronization.

Topics

• Asynchronous Batching
• LLM Inference Optimization
• CUDA Streams
• CUDA Events
• CUDA Graphs

Code references

• huggingface/blog
• huggingface/transformers
• Dao-AILab/flash-attention

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• AI in Multiple GPUs: Understanding the Host and Device Paradigm — Understanding Host-Device interaction, asynchronous execution, and CUDA Streams is fundamental for optimizing GPU performance.
• I Built a C++ Backend So My GPU Would Stop Eating Air — Optimizing LLM inference requires VRAM-aware bin packing and efficient host-to-device data transfer to eliminate padding overhead.
• Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization — LLMs can optimize GPU kernels by integrating with micro-profiling tools and iterative search, achieving significant speedups and energy efficiency.
• Every Python Concept a Generative AI Developer Actually Needs to Know — Mastering Python's concurrency, memory, and structural patterns is vital for building scalable and robust GenAI applications.
• Threshold-Based Exclusive Batching for LLM Inference — Optimal LLM inference batching depends on GPU bandwidth and workload, with a hybrid scheduler dynamically switching for superior throughput.
• Continuous Batching: How to Keep Your GPU Actually Busy — Continuous batching dynamically optimizes LLM inference throughput by keeping GPUs fully utilized through iteration-level scheduling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hugging Face - Blog.