Running a 35B LLM at 128K Context, Full Speed, on €870 of Used Hardware — No Cloud Required

2026-07-16 · Source: AI Advances - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Advanced, long

Summary

Benchmarking two used-GPU setups, a €600 build (20 GB VRAM) and a €1000 build (28 GB VRAM), revealed critical insights into local LLM inference performance. The €600 rig, combining an RTX 4070 (12 GB) and an RTX 2070 Super (8 GB), showed that active parameters, not total model size, dictate decode speed; for instance, a 26B MoE model achieved 80 tok/s, four times faster than a dense 27B model. Upgrading to the €1000 build, which swapped the 2070 Super for an RTX 5060 Ti (16 GB), increased VRAM capacity but reduced speed for smaller models by 7-8% due to the 5060 Ti's lower memory bandwidth. Crucially, the 35B MoE model, which previously spilled 30% to CPU, saw an 85% speed increase (43 to 79 tok/s) and could run at 104 tok/s with 128K context on tuned configurations, demonstrating the value of VRAM capacity for larger models.

Key takeaway

For MLOps Engineers building or upgrading local LLM inference hardware, prioritize VRAM capacity over raw GPU speed, especially when targeting 35B+ models or long contexts. Your €1000 budget is best spent on maximizing VRAM to avoid CPU spill, even if smaller models see a minor speed dip. Carefully configure `llama.cpp`'s `--tensor-split` and `--n-cpu-moe`, or Ollama's `num_gpu 99`, to optimize model placement and achieve full speed at 128K context.

Key insights

VRAM capacity, not raw speed, is key for large LLMs; active parameters determine decode throughput, often outperforming larger dense models.

Principles

• Active parameters, not total size, dictate LLM decode speed.
• VRAM capacity is paramount for long context and avoiding CPU spill.
• Memory bandwidth can limit decode speed, even with more VRAM.

Method

The article details how to configure `llama-server` with `--tensor-split`, `--n-cpu-moe`, and `--flash-attn on` for optimal multi-GPU LLM placement and performance, especially for MoE models. It also covers Ollama tuning.

In practice

• Select MoE models for better decode throughput on constrained VRAM.
• Configure `llama.cpp`'s `--n-cpu-moe` for spilling MoE models.
• Force Ollama layers onto GPU with `num_gpu 99` for models that fit.

Topics

• LLM Inference
• GPU Benchmarking
• VRAM Optimization
• Mixture-of-Experts
• llama.cpp
• Long Context LLMs

Code references

• ggml-org/llama.cpp
• ollama/ollama

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• A GPU-Poor’s Guide to Local LLM Inference in 2026 — Local LLM inference on consumer GPUs (4-12 GB VRAM) is now viable for long contexts via MoE models, advanced KV cache quantization, and CPU offloading.
• The Complete Guide to Running Local LLMs in 2026 — Self-hosting LLMs is now a practical choice, providing privacy, cost savings, and full data control over cloud alternatives.
• llama.cpp: Fast Local LLM Inference, Hardware Choices & Tuning — llama.cpp is the dominant open-source C/C++ framework enabling fast local large language model (LLM) inference, leveraging hardware advancements like NVIDIA's RTX 5090 and Apple's M4 Ultra for privacy…
• I Ran a 70B Language Model on My 4GB GPU and It Actually Worked — Here Is What I Found — AirLLM enables running 70B LLMs on 4GB GPUs without quantization, overcoming VRAM limitations.
• Fitting LLMs on Self-Hosted GPUs — VRAM capacity planning for LLMs relies on a formula considering precision, parameters, and KV cache multiplier.
• How To Reduce Inference Costs While Running LLMs — Optimizing LLM inference costs and performance requires a multi-faceted approach across hardware, software, and architectural techniques.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Advances - Medium.