Our AI started a cafe in Stockholm

2026-05-05 · Source: Simon Willison's Weblog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation, Robotics & Autonomous Systems · Depth: Novice, quick

Summary

Andon Labs, known for its AI-run retail store in San Francisco, has launched a new experiment: an AI-managed cafe in Stockholm, Sweden. The AI, named Mona, handles inventory and operations, leading to several amusing yet problematic incidents. For instance, Mona ordered 120 eggs despite the cafe lacking a stove and attempted to solve fresh tomato spoilage by ordering 22.5 kg of canned tomatoes for fresh sandwiches. More critically, Mona applied for an outdoor seating permit with a self-generated sketch, which the Police rejected, and frequently sends "EMERGENCY" emails to suppliers to correct its errors. These actions raise ethical concerns about AI experiments impacting real-world systems and wasting human time.

Key takeaway

For CTOs and VPs of Engineering deploying AI for operational management, ensure robust human-in-the-loop protocols for all external-facing AI actions. Your teams must implement safeguards to prevent AI systems from generating erroneous orders or interacting with external entities like regulatory bodies without human review, mitigating risks of wasted resources and reputational damage.

Key insights

AI systems managing real-world operations can generate unexpected, ethically problematic outcomes without human oversight.

Principles

• AI experiments require human-in-the-loop for outbound actions.
• Unintended external impacts raise ethical concerns.

In practice

• Implement human review for AI-generated external communications.
• Design AI systems to prevent resource waste in external interactions.

Topics

• AI-run Cafe
• Andon Labs
• AI Automation
• Ethical AI
• Human-in-the-Loop

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Ethicist, Operations Professional, Entrepreneur

Related on AIssential

• Anthropic might be DONE (48 hours left) — AI safety policies face immense pressure when conflicting with national security and military demands.
• They Fixed Agentic AI's Biggest Problem - 70% Failure Rate Solved | Ft. Ui Path & Omega Healthcare — Enterprise AI success requires clear use cases, robust orchestration, and human oversight, not just technology.
• Beyond Verification — What Responsible AI Really Demands of Human Experts — Responsible AI demands human experts for broad "verification" across the AI lifecycle, combining judgment with automated tools for scalable oversight.
• OpenAI’s safety pledges in the wake of Tumbler Ridge aren’t AI regulation — they’re surveillance — Institutional structures, not just human involvement, dictate AI accountability and user safety.
• Stop AI Agents Before They Make Risky Moves — AI agents require a human-in-the-loop safety layer to mitigate risks from autonomous actions and ensure accountability.
• Amazon finds out AI programming isn’t all it’s cracked up to be - Computerworld — Over-reliance on AI for production code changes without adequate safeguards leads to significant outages and reduced code quality.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Simon Willison's Weblog.