Viability of local models for coding

2026-07-07 · Source: Martin Fowler · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

An exploration of running local large language models (LLMs) for agentic coding on Apple Silicon machines (M3 Max 48GB RAM, M5 Pro 64GB RAM) reveals that while significant speed improvements have occurred, the overall quality for agentic coding remains inconsistent and not yet a "plug and play" experience. The author's four-week evaluation highlights RAM as the primary constraint, with models between 15-25GB being comfortable on 48GB RAM. Factors like processing power, memory bandwidth (~300 GB/s), model parameters (30B-80B), reasoning capabilities (sometimes counterproductive), and tool calling (tricky but self-correcting) all influence viability. Formats like GGUF/MLX and quantization (4BIT) were tested, with Mixture of Experts (MoE) architectures, specifically Qwen3.6 35B MoE, offering the best balance of capability, speed, and RAM footprint among those tried.

Key takeaway

For AI Engineers evaluating local LLMs for agentic coding, recognize that while models like Qwen3.6 35B MoE offer a good balance of capability and speed on Apple Silicon, a "plug and play" experience is not yet available. You should expect to invest time in tuning parameters like context window size (32K-64K) and experimenting with reasoning capabilities, as they can be counterproductive. Focus on MoE architectures to optimize RAM usage and performance.

Key insights

Local LLMs for agentic coding are improving but still require significant tuning and face quality limitations.

Principles

• RAM is the core constraint for local LLM runnability.
• MoE architectures balance parameters and RAM usage.
• Reasoning is not always necessary, can be counterproductive.

Method

The article describes an evaluation process involving running various quantized LLMs (Qwen3, Gemma 4) in GGUF/MLX formats on Apple Silicon, using LM Studio and coding harnesses like OpenCode and Pi, while adjusting context window sizes.

In practice

• Prioritize MoE models for better resource balance.
• Experiment with disabling reasoning for faster, better results.
• Set context window to 32K-64K for agentic coding.

Topics

• Local LLMs
• Agentic Coding
• Apple Silicon
• LLM Quantization
• Mixture of Experts
• Coding Harnesses
• LLM Performance

Code references

• earendil-works/pi
• anomalyco/opencode

Best for: AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• Usage-based pricing killing your vibe - here's how to roll your own local AI coding agents — Local LLMs offer a cost-effective alternative to commercial coding agents, leveraging specific models and optimized inference.
• A GPU-Poor’s Guide to Local LLM Inference in 2026 — Local LLM inference on consumer GPUs (4-12 GB VRAM) is now viable for long contexts via MoE models, advanced KV cache quantization, and CPU offloading.
• Qwen3.6 (27B) Local Test | Better than Gemma 4? | Coding, OCR, Images with llama.cpp | 🔴 Live — Qwen 3.6 (27B) offers strong reasoning and multimodal capabilities but suffers from slow local inference speed.
• How to Run LLMs Locally (Great For Learning and Privacy) — Specialized tools make running powerful LLMs locally feasible for privacy, learning, and production needs.
• Running local models is good now — Local LLMs have matured significantly, now enabling effective agentic coding and deep model introspection.
• 5x for Free : The Local Coding Stack — Local AI models, particularly MoE architectures, now offer competitive performance for coding with privacy and cost benefits.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Martin Fowler.