Fitting LLMs on Self-Hosted GPUs

2026-05-04 · Source: Anup Jadhav · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

A practical approach to estimating GPU VRAM requirements for deploying Large Language Models (LLMs) on self-hosted hardware is presented, designed to prevent Out-of-Memory (OOM) errors. The core formula, "vram_required = (bits_precision / 8) × params_billions × kv_cache_allocation", is central to capacity planning. The analysis highlights how storage precision, such as FP16, FP8, or FP4, drastically reduces VRAM needs; for instance, Llama 3 70B can go from 210 GB at FP16 to 52.5 GB at FP4 with a 1.5x multiplier. The "kv_cache_allocation" multiplier (1.5x to 2.5x) accounts for KV cache, activations, and runtime overhead, becoming critical for long contexts or large batch sizes. For Mixture of Experts (MoE) models like DeepSeek V3.1 (671B total, 37B active), VRAM planning for batched server inference must consider the total parameter count. The article also stresses aligning VRAM needs with available GPU "step-ladder" capacities (e.g., H100, H200, B200) and notes that embedding models require significantly less VRAM. A six-point checklist aids initial feasibility estimates.

Key takeaway

For MLOps Engineers deploying LLMs on self-hosted GPUs, accurately estimating VRAM is crucial to avoid costly OOM errors. You should apply the "vram_required" formula, carefully considering storage precision and the KV cache multiplier based on your workload. Always round your VRAM estimate up to the next available GPU step-ladder capacity, like H100 or B200, to prevent overspending on underutilized hardware. For MoE models, size based on total parameters for batched inference, not just active ones.

Key insights

VRAM capacity planning for LLMs relies on a formula considering precision, parameters, and KV cache multiplier.

Principles

• Model weights must reside in VRAM for inference.
• KV cache grows with batch size and sequence length.
• MoE VRAM sizing uses total parameters for batched inference.

Method

Estimate GPU requirements by determining bytes per parameter, total parameters, and a workload multiplier. Multiply these, then round up to the next GPU step-ladder rung, adding a 10-15% margin.

In practice

• Drop precision tiers (e.g., FP16 to FP8) to halve VRAM.
• Use 1.8x multiplier for production batched chat-style serving.
• Provision for GPU step-ladder rungs, not exact GB.

Topics

• LLM Inference
• GPU VRAM
• Model Quantization
• KV Cache
• Mixture-of-Experts
• Hardware Provisioning

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• A GPU-Poor’s Guide to Local LLM Inference in 2026 — Local LLM inference on consumer GPUs (4-12 GB VRAM) is now viable for long contexts via MoE models, advanced KV cache quantization, and CPU offloading.
• Running a 35B LLM at 128K Context, Full Speed, on €870 of Used Hardware — No Cloud Required — VRAM capacity, not raw speed, is key for large LLMs; active parameters determine decode throughput, often outperforming larger dense models.
• KV Cache Explained Like You’re an LLM Engineer — KV cache makes LLM inference viable by storing past Key/Value tensors, avoiding redundant attention recomputation.
• How I am Running Multiple LLMs on a Single H100 — Challenges and Resolution — Dynamic VRAM allocation and sequential booting are crucial for efficient multi-LLM deployment on single GPUs.
• I Deployed Local LLMs in Production for a Year. Part 2: The Operational Playbook — Optimizing local LLM deployments requires careful configuration of context, KV cache, and parallelism to balance memory, throughput, and latency.
• I Fine-Tuned an LLM on a 4GB Laptop GPU. Here’s the Roofline Math Big Labs Don’t Publish — LLM inference is fundamentally memory-bound, making memory bandwidth the critical bottleneck, not compute or VRAM capacity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Anup Jadhav.