Running an LLM Locally

2026-07-12 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Novice, short

Summary

An experiment explored running a Large Language Model (LLM) locally to mitigate the costs and usage limits associated with cloud-based AI for daily coding tasks. The author utilized CanIRun AI to identify compatible models, ultimately selecting Qwen 3.5 9B, a 9-billion-parameter model. Deployment was streamlined using Ollama, enabling a local setup without internet connectivity. While the initial setup was smooth, the local LLM exhibited slow inference speeds, with a 30-second response time for simple queries. When tested with basic coding tasks via OpenCode CLI, the model generated functional code but required significant cleanup and struggled with increased complexity, revealing a clear performance gap compared to larger cloud systems. Local models are deemed suitable for lightweight tasks and privacy-sensitive applications, but hardware limitations currently restrict their effectiveness for demanding development work.

Key takeaway

For software engineers evaluating LLM deployment strategies, understand that running models like Qwen 3.5 9B locally with tools like Ollama offers privacy and cost savings for lightweight tasks. However, be prepared for significant performance trade-offs and hardware bottlenecks, especially for complex coding or development work. Cloud models remain superior for demanding, high-context applications, but local options are improving for specific, less intensive use cases.

Key insights

Running local LLMs offers independence and privacy but trades raw power for hardware limitations.

Principles

• Local LLMs prioritize independence over raw power.
• Hardware significantly impacts local LLM performance.
• Privacy is a key advantage of local model execution.

Method

Use a tool like CanIRun AI to assess hardware compatibility for LLM selection, choose a suitable model (e.g., Qwen 3.5 9B), and deploy using a simplified runner like Ollama.

In practice

• Use local LLMs for quick questions or summaries.
• Test small coding ideas without API costs.
• Prioritize local execution for sensitive data.

Topics

• Local LLMs
• Ollama
• Qwen 3.5 9B
• AI Hardware
• Cloud AI Costs
• Code Generation

Best for: AI Engineer, Software Engineer, AI Student

Related on AIssential

• 5 AI Tools That Let Me Run LLMs Locally (No API Costs) — Running LLMs locally is now a mature, cost-effective, and private alternative to API-based solutions.
• What is your current local LLM setup? — The community actively explores diverse local LLM setups, balancing hardware, software, and model choices for varied applications.
• Setting Up Your Own Large Language Model — Running capable LLMs locally ensures privacy and digital sovereignty, reducing reliance on cloud services.
• How to Run LLMs Locally (Great For Learning and Privacy) — Specialized tools make running powerful LLMs locally feasible for privacy, learning, and production needs.
• Building an Agentic AI App With a Local LLM — No Cloud, No API Costs — Local LLMs on consumer GPUs enable cost-free, private AI app development with cloud-ready architecture.
• Best Local AI Coding Model? — The optimal AI coding workflow integrates local models for daily tasks and cloud models for complex challenges.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.