Andrej Karpathy Automated His Home

2026-03-21 · Source: No Priors: AI, Machine Learning, Tech, & Startups · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Internet of Things (IoT) & Connected Devices, Robotics & Autonomous Systems · Depth: Novice, quick

Summary

Andrej Karpathy has developed a personal home automation system, dubbed "Dobby the elf claw," utilizing AI agents to manage various smart home subsystems. The system autonomously discovered devices like Sonos speakers and lighting controls on his local network, reverse-engineered their APIs through web searches, and then created a unified dashboard. This setup allows Karpathy to issue natural language commands, such as "Dobby at sleepy time," to control lights, HVAC, shades, pool, spa, and security systems. The project demonstrates the practical application of AI agents in integrating disparate smart home technologies and creating a centralized control interface.

Key takeaway

For AI Architects or AI Engineers exploring practical applications of agentic AI, consider how you might apply similar autonomous discovery and API integration techniques to complex, multi-vendor environments. Your team could develop custom, unified control systems for internal infrastructure or specialized IoT deployments, moving beyond off-the-shelf solutions to create highly tailored and efficient management interfaces.

Key insights

AI agents can autonomously discover, integrate, and control diverse smart home devices via natural language.

Principles

• AI agents can reverse-engineer APIs.
• Local network discovery is effective for smart devices.

Method

The method involves AI agents performing local network scans, web searches for API documentation, and then generating control interfaces and dashboards for discovered smart home devices.

In practice

• Automate home lighting with voice commands.
• Integrate disparate smart home systems.
• Create custom control dashboards.

Topics

• Home Automation
• AI Agents
• Smart Home Systems
• API Discovery
• Local Area Network

Best for: AI Architect, AI Engineer, Entrepreneur, AI Chatbot Developer, AI Product Manager, Software Engineer

Related on AIssential

• Agentic AI for Manufacturing: Fragmented Data & LLM Reasoning Challenges — Agentic AI enhances industrial IoT by enabling natural language querying and autonomous reasoning over fragmented sensor data.
• Building Agents at Home: Parenting, Work, and Benevolent Neglect — AI agents can automate complex personal and professional tasks, enabling individuals to pursue ambitious goals despite significant time constraints.
• Machine-to-machine payments unlock ephemeral, API-first businesses — AI agents can autonomously plan events by integrating with real-world third-party services for research and transactions.
• Andrej Karpathy's new open source 'autoresearch' lets you run hundreds of AI experiments a night — with revolutionary implications — AI agents can autonomously conduct and optimize experiments, accelerating research and discovery across diverse fields.
• Karpathy: Stop Building for Humans — Future device interaction should prioritize agents using APIs over humans using individual apps.
• Building Intelligent Research Agents with Manus - Ivan Leo, Manus AI (now Meta Superintelligence) — Manis provides a general AI agent and API for automating complex tasks and workflows across diverse platforms.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by No Priors: AI, Machine Learning, Tech, & Startups.