Setting Up Your Own Large Language Model

2026-07-04 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

This guide details setting up a local large language model, specifically Qwen 3 8B, on an Apple Silicon Mac, such as a MacBook Air M4 with 24 GB unified memory. The process leverages Ollama, an open-source framework, to run the 9-billion-weight model, which occupies approximately 6 GB of RAM and requires a 5.2 GB download. The article emphasizes the benefits of local LLMs for digital sovereignty and privacy, allowing offline operation without data leaving the machine, contrasting this with cloud-based alternatives. It provides a step-by-step installation using terminal commands, covers interactive chat, one-shot commands, and HTTP API interaction, and explains how to manage "thinking" tokens. Additionally, it outlines integration with VS Code via the Continue.dev extension for an offline coding assistant.

Key takeaway

For AI Engineers or developers handling sensitive data, deploying local LLMs like Qwen 3 8B via Ollama on Apple Silicon Macs offers critical digital sovereignty. You can ensure data privacy by keeping all interactions offline, avoiding external API costs and data retention policies. Consider this approach for applications requiring strict confidentiality or offline functionality, and explore integrating it with tools like VS Code for a secure, personalized AI assistant.

Key insights

Running capable LLMs locally ensures privacy and digital sovereignty, reducing reliance on cloud services.

Principles

• Local LLMs enhance digital sovereignty and privacy.
• Unified memory significantly boosts local LLM performance.
• Open-source models are rapidly closing the performance gap.

Method

Install Ollama, a single binary framework, then use its CLI to pull a desired open-source model like Qwen 3 8B. Start the Ollama server and interact via chat, terminal commands, or HTTP API.

In practice

• Use Ollama for simplified local LLM deployment.
• Configure VS Code with Continue.dev for offline coding.
• Disable "thinking" tokens for faster responses.

Topics

• Local LLMs
• Ollama
• Apple Silicon
• Digital Sovereignty
• Qwen 3 8B
• VS Code Integration
• Open-Source AI

Best for: Machine Learning Engineer, AI Engineer, Software Engineer

Related on AIssential

• Building an Agentic AI App With a Local LLM — No Cloud, No API Costs — Local LLMs on consumer GPUs enable cost-free, private AI app development with cloud-ready architecture.
• How to Run LLMs Locally (Great For Learning and Privacy) — Specialized tools make running powerful LLMs locally feasible for privacy, learning, and production needs.
• 5 Cool Things I Did with Local Language Models — Local LLMs offer superior privacy, cost, and offline capabilities for sensitive data and proprietary workflows.
• Ollama: Unleash the Power of Local LLMs on Your Machine — Ollama enables private, cost-effective, and low-latency local LLM inference and customization via a Python library.
• Using Local Coding Agents — Local coding agents offer privacy, cost control, and reproducibility using open-weight LLMs and customizable harnesses.
• What is your current local LLM setup? — The community actively explores diverse local LLM setups, balancing hardware, software, and model choices for varied applications.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.