Running Multiple Models on One GPU with vLLM and GPU Memory Utilization

2026-04-02 · Source: Andrej Baranovskij · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, quick

Summary

The provided content details a method for controlling GPU memory allocation when running large language model (LLM) inference using the vLLM framework on Linux with Nvidia GPUs. The author, Andre, utilized a 96 GB Nvidia GPU for Sparrow, an open-source solution for structured data extraction. The challenge involved loading two different models, Mistral small 3.2 (24 billion parameters) and a smaller dots OCR model, simultaneously into memory to avoid slow loading times during sequential inference requests. vLLM offers a `gpu_memory_utilization` parameter, allowing users to predefine the percentage of GPU memory allocated to each model. For instance, 70% was allocated to Mistral and 20% to dots OCR, leaving 10% free. This functionality enabled both models to be loaded, cached, and kept in memory for efficient inference serving.

Key takeaway

For AI Engineers managing multiple LLMs on a single Nvidia GPU, leveraging vLLM's `gpu_memory_utilization` parameter is crucial. This allows you to pre-allocate specific memory percentages for each model, ensuring both are loaded and cached simultaneously. This approach significantly reduces model loading overhead between inference requests, optimizing throughput and responsiveness for diverse use cases without requiring additional hardware.

Key insights

vLLM's `gpu_memory_utilization` parameter enables efficient multi-model inference on a single GPU by pre-allocating memory.

Principles

• Caching models in memory improves inference speed.
• Pre-allocating GPU memory prevents OOM errors.

Method

Use vLLM's `gpu_memory_utilization` parameter to specify memory percentages for each model, allowing multiple models to reside in GPU memory simultaneously for faster inference.

In practice

• Allocate 70% for a 24B parameter model.
• Allocate 20% for a smaller OCR model.

Topics

• vLLM Framework
• GPU Memory Allocation
• Large Language Model Inference
• NVIDIA GPU
• Mistral Model

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• Maximizing GPU Utilization with NVIDIA Run:ai and NVIDIA NIM — Intelligent orchestration with NVIDIA NIM and Run:ai significantly boosts GPU utilization and inference performance for LLMs.
• I Built a C++ Backend So My GPU Would Stop Eating Air — Optimizing LLM inference requires VRAM-aware bin packing and efficient host-to-device data transfer to eliminate padding overhead.
• Running 26B and 35B LLMs at Full Speed on €990 of Used Hardware — No Cloud Required — Cost-effective local LLM inference is achievable with dual secondhand GPUs, outperforming single high-end cards and revealing speculative decoding's limitations on PCIe.
• How I am Running Multiple LLMs on a Single H100 — Challenges and Resolution — Dynamic VRAM allocation and sequential booting are crucial for efficient multi-LLM deployment on single GPUs.
• Sparser, Faster, Lighter Transformer Language Models — Reshaping LLM sparsity to fit GPU architecture significantly improves inference and training speed.
• Optimizing Local LLM Inference on Constrained Hardware — Bypassing high-level LLM wrappers and direct llama.cpp tuning doubles performance on constrained hardware.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Andrej Baranovskij.