Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption

2026-05-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Expert, extended

Summary

An empirical study investigates multi-model Large Language Model (LLM) scheduling challenges on shared, heterogeneous hardware, focusing on CPU-GPU offloading and preemption. The research, conducted on NVIDIA RTX 5000 Ada Generation (32 GB VRAM) and RTX A6000 (48 GB VRAM) GPUs with an AMD Threadripper PRO 5995WX CPU, analyzed Llama 3 8B, Qwen3-32B, Llama 2 70B (for offloading in Q4 format) and Qwen2.5-3B, Qwen3-8B, Qwen2.5-14B (for preemption in FP16). Findings reveal that offloading causes non-linear, model-dependent decode throughput degradation, with smaller models showing sharper sensitivity. Preemption overhead, ranging from approximately 2.6 s to 7.3 s, is overwhelmingly dominated by model weight reload (over 98.5%) rather than KV cache transfer, and this cost is largely constant regardless of preemption point. The study identifies critical features for next-generation schedulers, including model-specific offloading sensitivity, workload characteristics, and hardware-aware preemption cost structures.

Key takeaway

For AI Architects designing multi-model LLM serving systems, prioritize hardware-aware scheduling that accounts for model-specific offloading sensitivities. Your system should treat preemption costs as a fixed penalty per model and hardware pair, as model reload dominates overhead, not KV cache transfer. This allows for more predictable resource allocation and efficient management of heterogeneous workloads, even under memory constraints.

Key insights

Multi-model LLM schedulers must account for non-linear offloading sensitivity and model reload-dominated preemption costs.

Principles

• LLM offloading sensitivity is non-linear and model-dependent.
• Preemption overhead is fixed, dominated by model reload.
• Hardware characteristics significantly influence scheduling factors.

Method

The study empirically evaluated LLM behavior by sweeping GPU layer allocation for offloading and instrumenting the preempt-resume cycle, measuring decode throughput and overhead decomposition across diverse models and hardware.

In practice

• Model-specific offloading profiles are crucial for placement.
• Treat preemption cost as a fixed penalty per model/hardware.
• Consider expected output length for resource allocation.

Topics

• LLM Inference
• Multi-Model Scheduling
• CPU-GPU Offloading
• Preemption Costs
• KV Cache Management
• Heterogeneous Hardware

Best for: MLOps Engineer, AI Scientist, Research Scientist, Machine Learning Engineer, AI Engineer, AI Architect

Related on AIssential

• Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption — Multi-model LLM serving requires schedulers to account for non-linear offloading sensitivity and high preemption costs across diverse models and hardware.
• What I Learned From Implementing LLM Architectures From Scratch (And How to Get Started) — Coding LLM architectures from scratch reveals critical design nuances and optimization strategies for memory and performance.
• How I am Running Multiple LLMs on a Single H100 — Challenges and Resolution — Dynamic VRAM allocation and sequential booting are crucial for efficient multi-LLM deployment on single GPUs.
• An Interpretable Latency Model for Speculative Decoding in LLM Serving — A new latency model explains speculative decoding performance in LLM serving under dynamic loads.
• Prefill/Decode Disaggregation: Why Your GPU Can’t Do Two Things at Once — LLM inference's prefill and decode phases have conflicting hardware demands, necessitating disaggregation for optimal performance and user experience.
• CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation — Disaggregating KV-cache and FFN weights into separate GPU pools optimizes multi-LLM serving for cold MoE models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.