I Ran Every Local LLM on a 24GB GPU So You Don’t Have To

2026-07-26 · Source: Artificial Intelligence on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, long

Summary

An extensive three-month analysis evaluated 15 local Large Language Models (LLMs) on a single 24GB RTX 3090 GPU, using llama.cpp with Q4_K_M quantization. The study found that smaller models like Gemma 4 16B offer high speed (85 tok/s) for specialized tasks, while mid-range models such as Qwen 3.6 35B provide GPT-3.5-level quality at usable speeds (32 tok/s), requiring dedicated GPU resources. Larger models like DeepSeek V4-Flash 72B achieve near GPT-4 quality but at slower rates (14 tok/s) even with hybrid inference. Cost analysis indicates local LLMs are 10x cheaper for heavy users (>\$225/month API vs. ~\$24/month local), with a breakeven point around 100K output tokens/month. The report highlights setup complexities, GPU memory management, and the need for a single-purpose machine during inference, recommending specific models for different use cases.

Key takeaway

For AI Engineers evaluating local LLM deployments on consumer hardware, understand that a 24GB GPU like the RTX 3090 offers a sweet spot for performance and cost. Prioritize models like Qwen 3.6 35B for balanced quality and speed, or DeepSeek Coder V2 Lite 16B for coding. Be prepared for your machine to become single-purpose during inference and factor in the initial setup time, as API costs remain more economical for low-volume usage.

Key insights

Consumer GPUs like the RTX 3090 now enable practical local LLM inference, balancing cost, speed, and quality for various use cases.

Principles

• Small LLMs excel as specialists but lack general reasoning depth.
• 30-40B parameter models offer the best local quality-speed balance.
• Local LLM cost savings are significant only for high-volume usage.

Method

Models were tested using llama.cpp with Q4_K_M quantization on a single RTX 3090 24GB, measuring tokens/second and grading output quality across technical, creative, and summary tasks.

In practice

• Use Gemma 4 16B for fast, general assistant tasks.
• Employ DeepSeek Coder V2 Lite 16B for reliable code generation.
• Run Qwen 3.6 35B for GPT-3.5 quality on a single 24GB card.

Topics

• Local LLMs
• GPU Inference
• RTX 3090
• Model Quantization
• Performance Benchmarking
• LLM Cost Analysis
• DeepSeek V4-Flash

Best for: Machine Learning Engineer, AI Engineer, Director of AI/ML

Related on AIssential

• Best Local LLM for Coding in 2026? — Optimal local coding LLM selection prioritizes hardware fit and practical performance over raw benchmark scores.
• Building a monokernel for LLM inference on AMD MI300X - up to 3,300 output tokens/s per request [P] — Optimizing LLM inference on AMD MI300X by leveraging hardware die topology with a single GPU-resident monokernel.
• A GPU-Poor’s Guide to Local LLM Inference in 2026 — Local LLM inference on consumer GPUs (4-12 GB VRAM) is now viable for long contexts via MoE models, advanced KV cache quantization, and CPU offloading.
• LLM Lab Setup and Hardware Requirements (Part 2) — Local LLM deployment requires careful hardware and model selection for successful operation.
• Can an RTX 4060 Laptop Replace Cloud LLMs for Everyday Coding? — Hardware efficiency and context window optimization are crucial for practical local LLM coding workflows.
• I Fine-Tuned an LLM on a 4GB Laptop GPU. Here’s the Roofline Math Big Labs Don’t Publish — LLM inference is fundamentally memory-bound, making memory bandwidth the critical bottleneck, not compute or VRAM capacity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence on Medium.