New AI agent learns to use CAD to create 3D objects from sketches

2025-11-19 · Source: MIT News - automation · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Intermediate, medium

Summary

MIT engineers have developed a new AI model that can operate Computer-Aided Design (CAD) software to create 3D objects from 2D sketches, aiming to simplify the complex CAD learning curve. Published on November 19, 2025, this system functions like a human user, executing keyboard and mouse commands and selecting software options. The team created a dataset called VideoCAD, comprising over 41,000 examples of 3D model construction in CAD, detailing specific clicks, mouse-drags, and keyboard actions. This dataset trains the AI to translate high-level design commands into precise UI interactions. The project envisions an AI-enabled "CAD co-pilot" that could automate tedious tasks, suggest next steps, and enhance productivity for engineers while making CAD more accessible to a broader audience. The work will be presented at NeurIPS in December.

Key takeaway

For AI Scientists developing intelligent agents for complex software, this research demonstrates a viable approach to training models on detailed user interface interactions. You should consider creating specialized datasets that map high-level tasks to granular UI actions to enable agents to operate software like a human. This method could significantly expand the capabilities of AI co-pilots in professional applications.

Key insights

An AI agent can learn to operate complex CAD software by observing human UI interactions from a specialized video dataset.

Principles

• UI agents can automate complex software tasks.
• Detailed UI interaction data is crucial for AI training.
• AI can lower barriers to entry for specialized software.

Method

The team developed VideoCAD, a dataset of 41,000+ videos translating high-level CAD commands into specific UI actions (clicks, drags). This data trains an AI model to directly control CAD software from 2D sketches.

In practice

• Automate repetitive CAD modeling work.
• Assist new users in learning CAD software.
• Generate 3D models from 2D sketches.

Topics

• AI Agents
• Computer-Aided Design
• 3D Modeling
• VideoCAD Dataset
• User Interface Automation

Best for: AI Scientist, AI Engineer, Machine Learning Engineer, Research Scientist

Related on AIssential

• VISUALSKILL: Multimodal Skills for Computer-Use Agents — Multimodal skills, integrating visuals with text, significantly improve computer-use agents' performance on GUI tasks.
• Your Path to Agentic AI for Production — The course teaches practical AI agent development, focusing on building production-ready multi-agent systems.
• Transcript: 'Inside OpenAI's Agentic Browser, Atlas' — Agentic browsers integrate AI directly into web navigation, automating tasks and providing contextual assistance.
• this is the ONLY AI skill you need to have (seriously) — AI agents will simplify technology use for most people by consolidating UIs into a single conversational interface.
• 🎧 Inside OpenAI’s Agentic Browser, Atlas — OpenAI's Atlas browser integrates AI agents to automate web tasks, enhancing user experience and developer velocity.
• What We Learned Building an AI Agent for 3D Creation — Meshy, a 3D generation platform, released an article on June 17th, 2026, titled "What We Learned Building an AI Agent for 3D Creation." The platform's core function involves rapidly transforming text…

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by MIT News - automation.