Agents on the Canvas in tldraw — Steve Ruiz, tldraw

2026-05-01 · Source: AI Engineer · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Intermediate, long

Summary

Steve Ruiz from Teal Draw introduced the company's online whiteboard, SDK, and its applications, particularly in AI-driven development. Teal Draw's SDK is used in products like Replit's agent canvas and Luma AI's canvas. Ruiz demonstrated "Make Real," an early 2023 project that converted canvas drawings into working prototypes using vision models, enabling non-technical users to create technical outputs. He then showcased Teal Draw's evolution, starting with AI as a collaborator on the canvas for tasks like drawing diagrams or completing slides using structured text outputs. The presentation advanced to an "agentic loop" for iterative design, and finally, to "fairies"—multiple, visible agents on the canvas that can collaborate, delegate tasks, and even orchestrate complex workflows like generating wireframes from text descriptions. The latest development involves a desktop app that allows AI agents to directly run JavaScript against the editor, enabling deep interactivity and local file manipulation, pushing the boundaries of user agency and local-first applications.

Key takeaway

For CTOs and VPs of Engineering exploring advanced AI integration, consider how direct, visual agent interaction within a canvas environment can transform prototyping and development workflows. Your teams could leverage local-first desktop applications to grant AI agents deeper, sandboxed access to system APIs, enabling unprecedented levels of automation and interactivity for tasks like code generation, UI manipulation, and even system-level scripting, albeit with careful consideration of security implications.

Key insights

AI agents can evolve from single-shot generation to collaborative, orchestrated entities directly interacting with canvas environments.

Principles

• AI as collaborator, not just generator
• Visualizing agent state aids understanding
• Local-first apps enhance AI agency

Method

An agentic loop involves AI producing an output, reviewing it, and iterating until a task is deemed complete, often with sub-features like rejection and progress visualization.

In practice

• Use structured text outputs for AI-generated diagrams
• Implement multi-agent systems for complex tasks
• Explore local-first apps for enhanced AI scripting

Topics

• tldraw SDK
• AI Agents
• Canvas-based AI
• Agent Orchestration
• Vision Models

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Engineer, Software Engineer, AI Architect

Related on AIssential

• From agents to world models: What San Francisco revealed about AI’s next phase — AI is transitioning from linguistic models to agentic systems that act, use tools, and comprehend physical and virtual environments.
• Build Interactive Agents with Generative UI — AI agents are evolving from text-only outputs to generating interactive, custom user interfaces.
• AgentCraft: Putting the Orc in Orchestration — Ido Salomon — Effective multi-agent orchestration requires enhanced visibility, delegated autonomy, and seamless human-agent collaboration.
• The Death of the “Everything Prompt”: Google’s Move Toward Structured AI — Google's Interactions API provides structured state management and asynchronous agent orchestration for complex AI applications.
• BuilderIO / agent-native — The Agent-Native framework unifies autonomous agents and UIs as equal system components, enabling collaborative, context-aware, and self-improving applications.
• From Dashboards to Decisions: How We’re Building AI Agents at RaySuite AI — AI agents transform user interaction by providing intelligent, contextual answers and taking action over complex systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Engineer.