I Built a $0/Month Jarvis With an AI Pair Programmer, and Everything That Broke Along the Way

2026-07-19 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

A developer successfully built Sorena, a local-first, \$0/month voice assistant running on a Windows machine, using Claude Code as an AI pair programmer. The project, detailed across seven phases, emphasized a "loop engineering" workflow within spec-driven development, supported by Architecture Decision Records (ADRs). Key technical implementations include a LiteLLM-based router for Groq and Gemini free tiers, a hand-coded ReAct agent loop, and a multi-agent architecture. Significant challenges involved improving speech-to-text accuracy from 40% to 100% by switching to WASAPI audio, tuning openWakeWord's `hey_jarvis` model, and building long-term memory with SQLite and `all-MiniLM-L6-v2` embeddings. The project also developed a robust 23-case eval suite and revealed that model validation metrics (e.g., 0.468 recall for a custom wake word) are often a ceiling, not a real-world promise.

Key takeaway

For AI Engineers building local-first agent systems, prioritize foundational engineering practices over immediate framework adoption. Your validation metrics are a ceiling, not a guarantee; rigorously measure performance on real hardware and data. Build core agent loops manually once to ensure deep understanding and debuggability. Document every architectural decision and bug as an ADR to create a project memory and regression suite, ensuring long-term stability and maintainability.

Key insights

Real-world AI system development demands rigorous measurement, spec-driven workflows, and deep understanding of underlying components, often revealing validation metric limitations.

Principles

• Measure against deployed capture paths.
• Build agent loops raw to understand them.
• ADRs capture critical design decisions.

Method

The "loop engineering" workflow involves writing unambiguous specs, scheduling autonomous overnight builds, and conducting adversarial review passes before release tagging.

In practice

• Use `uv`, `ruff`, `pytest` for robust repo foundation.
• Implement multi-provider LLM routing with exponential backoff.
• Hand-code ReAct loops for debuggability.

Topics

• AI Agents
• Voice Assistants
• LLM Engineering
• MLOps
• Spec-Driven Development
• Architecture Decision Records
• Wake Word Detection

Code references

• BerriAI/litellm
• jd/tenacity
• modelcontextprotocol/python-sdk
• SYSTRAN/faster-whisper
• dscripka/openWakeWord

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Hard Lessons Building Agents Since GPT-3.5 — Agent development demands a mindset shift from deterministic code preservation to embracing rapid iteration and model empathy.
• Sixteen Claude Agents Built a C Compiler Without Human Intervention... Almost — Autonomous AI agents can collaboratively build complex software like a C compiler with minimal human oversight.
• Before You Ship Your Agent: A Five-Step Path to Evaluations You Can Trust — Agent evaluation requires session-level metrics and validating the evaluation tools themselves for trustworthy results.
• Buffett Had Moody’s Manuals. I Built a Desk of AI Analysts. — Building trustworthy multi-agent AI systems requires rigorous engineering, particularly robust validation, retry logic, and honest evaluation.
• How to Build Your Operating System Using Codex, for Beginners — Effective AI agent use hinges on configuring four core components: Identity, Tools, Skills, and Memory, plus user trust.
• Introducing the agent performance loop: AgentCore Optimization now in preview — AgentCore Optimization introduces a data-driven loop for continuous AI agent improvement, replacing manual debugging.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.