Developing Offline AI Agents: Local RAG and Microsoft Foundry Local (Part 2)

2026-07-19 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, short

Summary

This article, "Developing an Offline AI Agent: Local RAG and Microsoft Foundry Local (Part 2)," details the project development phase for creating a local AI assistant. It addresses the challenge of large text files exceeding AI model context windows and impacting performance, proposing a "chunking and retrieval" strategy. The process involves using `ingest.py` to divide "Cybersecurity Club Notes" into paragraphs, converting them into 1024-dimensional vectors via Foundry Local's `qwen3-embedding-0.6b` model, and storing them in an SQLite database. Subsequently, `retrieve.py` employs cosine similarity to match user queries, also vectorized, with the most relevant text chunks, demonstrating a 0.6261 similarity score for a training query. Finally, `app.py` integrates the `qwen3` search engine and `phi-3.5-mini` chat model into a command-line interface, enabling the agent to provide accurate, non-hallucinatory Turkish answers based solely on retrieved local data.

Key takeaway

For AI Engineers developing local, privacy-focused applications, implementing a RAG architecture with chunking and cosine similarity is crucial. Your models will avoid context window limits and reduce computational load, ensuring faster, more stable responses without cloud data transfer. Consider using local embedding models like `qwen3-embedding-0.6b` and chat models such as `phi-3.5-mini` to build efficient, offline agents. This approach enhances privacy and performance significantly.

Key insights

Offline AI agents can achieve fast, accurate, and private responses by combining local RAG with efficient chunking and retrieval.

Principles

• AI models have context window limits.
• Large inputs exhaust local device resources.
• Cosine similarity enables semantic search.

Method

Divide large texts into paragraphs (chunks), vectorize them using an embedding model like `qwen3-embedding-0.6b`, store in a local database, then retrieve relevant chunks via cosine similarity for a chat model like `phi-3.5-mini` to answer.

In practice

• Use `text.split('\n\n')` for paragraph chunking.
• Store embeddings in a serverless SQLite database.
• Integrate embedding and chat models via CLI.

Topics

• Local RAG
• Offline AI Agents
• Text Chunking
• Cosine Similarity
• Microsoft Foundry Local
• Qwen3 Embedding
• Phi-3.5-mini

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• AI tips & tricks (Ep. 307) — RAG systems enable secure, cost-effective, and performant AI for institutional knowledge by localizing data and leveraging smaller models.
• How to Chat With Your Codebase Locally and Privately, No Code Leaves Your Machine — Local, private AI code assistants prevent hallucinations and data leaks by indexing code with structure-aware chunking.
• AI for Developers - Cohere — Cohere's "AI for Developers" blog highlights recent advancements and practical guides across various AI domains, with articles published from October 2024 through May 2026.
• The Road to Unlimited Agent Memory — No RAG, No Papers, Just Markdown — Unlimited agent memory can be achieved simply using plain markdown files, bypassing complex RAG systems.
• Your Local AI Is Dumb. Not Because of the Model. Because of What It Can’t See. — RAG and MCP transform generic local AI into a powerful, private, business-aware assistant.
• Proxy-Pointer RAG: Multimodal Answers Without Multimodal Embeddings — Multimodal RAG success hinges on aligning retrieval with document structure, not just embedding similarity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.