Experiences with local models for coding

2026-07-08 · Source: Martin Fowler · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

This article details experiences running small language models locally for agentic coding on M3 Max (48GB RAM) and M5 Pro (64GB RAM) machines. The author evaluated models like Qwen3.6 35B MoE, Gemma 4 31B, and Gemma 4 26B through a viability funnel, manual and automated evaluations, and day-to-day use. Key findings include Qwen Coder Next 80B MoE's functional correctness but instability, and Qwen3.6 35B MoE's mixed results, often struggling with complex tasks like cumulative percentages or creating new charts from log data. Automated evaluations sometimes contradicted manual findings, with Qwen 35B MoE performing better on a 64GB machine than 48GB for one task. The author notes that task complexity, file discovery, and instruction specificity significantly impact success, finding Bash and Python tasks generally more successful than JavaScript.

Key takeaway

For AI Engineers evaluating local LLMs for agentic coding, recognize that current small models like Qwen3.6 35B MoE are not plug-and-play solutions. You should carefully select tasks, prioritizing small, well-defined changes over complex, multi-file operations requiring extensive code discovery. Be prepared for significant configuration effort and rigorous code review, as output quality and stability can vary even with identical model settings across different hardware.

Key insights

Local small models for agentic coding are not plug-and-play, requiring careful task selection and setup.

Principles

• Viability follows a funnel: RAM, speed, tool calling, correctness, context, complexity, quality.
• Task characteristics (discovery, file count, specificity, tech stack) are critical for local model success.
• Automated evaluations may not always confirm manual user experience.

Method

Evaluate local models through a three-phase journey: manual user experience evaluations, automated one-shot problem-solving, and integration into daily workflows for real-world tasks.

In practice

• Prioritize small, straightforward tasks, often pre-planned by larger models.
• Use OpenCode or Pi as coding harnesses.
• Disable "Enable Thinking" in LM Studio for Qwen3.6 35B MoE.

Topics

• Local LLMs
• Agentic Coding
• Model Evaluation
• Qwen3.6
• Gemma 4
• Coding Harnesses
• Developer Workflows

Code references

• birgitta410/local-coding-evals
• Egonex-AI/Understand-Anything

Best for: AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• Quoting Georgi Gerganov — The Qwen3.6-27B model is highly effective for local coding tasks, even for mundane developer workflows.
• 5x for Free : The Local Coding Stack — Local AI models, particularly MoE architectures, now offer competitive performance for coding with privacy and cost benefits.
• Hermes Agent with Qwen3.6 (Local) | AI App Template with LangChain, LangGraph, llama.cpp | 🔴 Live — Local LLMs like Qwen 3.6 face significant challenges in agentic coding tasks, especially with lower quantization.
• [AINews] Top Local Models List - April 2026 — Community consensus on local LLMs prioritizes practical usability and specific use cases over raw benchmark supremacy.
• GLM 5.1 Agentic Coding Test with OpenCode | New Best Open Coding LLM? | Live Test — GM 5.1 excels in agentic coding and project understanding, outperforming some established models in specific benchmarks.
• Using Local Coding Agents — Local coding agents offer privacy, cost control, and reproducibility using open-weight LLMs and customizable harnesses.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Martin Fowler.