Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning

2026-06-12 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Expert, extended

Summary

Prism is a multi-LLM serving system designed to enhance cost efficiency and Service Level Objective (SLO) attainment by optimizing GPU sharing. It addresses limitations in existing systems that struggle with dynamic workloads and lack cross-model memory coordination. Prism achieves this through two core designs: on-demand memory allocation, which dynamically maps physical to virtual memory pages for flexible redistribution, and a two-level scheduling policy that adjusts sharing strategies based on runtime demands. Evaluations using real-world traces show Prism delivers over 2x cost savings and 3.3x SLO attainment compared to state-of-the-art systems like MuxServe++ and static partitioning, supporting up to 3.5x more requests. It activates 8B models in 0.7s and 70B models in 1.5s.

Key takeaway

For MLOps Engineers managing multi-LLM inference infrastructure, consider adopting systems like Prism to significantly reduce operational costs while meeting stringent latency SLOs. Its dynamic memory coordination and two-level scheduling approach enable efficient GPU utilization, preventing underutilization during idle periods and adapting to rapid workload fluctuations. Implementing on-demand memory allocation and SLO-aware scheduling can yield substantial cost savings and improve overall service reliability for diverse LLM deployments.

Key insights

Dynamic, demand-aware cross-model memory coordination is crucial for cost-efficient multi-LLM serving with strict SLOs.

Principles

• Combine space and time GPU sharing flexibly.
• Dynamically adjust resource allocation to workload.
• Prioritize resource allocation by SLO diversity.

Method

Prism uses a kvcached shim layer for on-demand memory allocation, decoupling virtual and physical GPU memory. It employs a two-level scheduler: a global scheduler for model-to-GPU placement using KV pressure ratio, and a local scheduler for priority-based request dispatch.

In practice

• Implement on-demand memory allocation for KV caches.
• Use a two-level scheduling for LLM placement.
• Prioritize requests by TTFT SLOs.

Topics

• LLM Serving
• GPU Sharing
• Memory Management
• SLO Attainment
• Dynamic Scheduling
• KV Cache

Code references

• AliyunContainerService/gpushare-scheduler-extender
• QLM-project/QLM

Best for: AI Architect, NLP Engineer, CTO, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Serving — Optimizing CPU LLM serving requires separate execution plans for prefill and decode phases due to their distinct computational and memory demands.
• Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference — Optimizing LLM inference by routing requests to specialized short or long context pools significantly reduces GPU costs and improves reliability.
• Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient — Continuous batching optimizes LLM inference by dynamically scheduling requests and packing tokens to eliminate padding and idle GPU time.
• PALS: Power-Aware LLM Serving for Mixture-of-Experts Models — PALS optimizes LLM serving energy efficiency by dynamically controlling GPU power caps and batch sizes using feedback-driven models.
• How I am Running Multiple LLMs on a Single H100 — Challenges and Resolution — Dynamic VRAM allocation and sequential booting are crucial for efficient multi-LLM deployment on single GPUs.
• assigning Moe to Gpus to reduce inference and memory usage — Optimizing MoE expert-to-GPU assignment using training logs is a critical challenge for LLM inference efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.