Merging design and computer science in creative ways

2025-04-28 · Source: MIT News - Natural language processing · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Advanced, medium

Summary

MIT MAD Fellow Alexander Htet Kyaw, a dual master's student in architectural studies and electrical engineering, integrates AI, augmented reality (AR), and robotics with gesture and speech recognition to create human-AI workflows. His projects include Curator AI, an AR-powered platform that won $26,000 in OpenAI products and cash at an MIT hackathon, designed to streamline online furniture shopping by providing context-aware recommendations based on room dimensions and user prompts. Another project, Estimate, uses AR and object-recognition AI to generate detailed cost estimates for renovations and paint jobs for small businesses, earning $5,000. Kyaw also developed Unlog, an AR and gesture recognition system for mapping building component dimensions to construct sustainable structures from whole logs, and BendShelters, which uses physics simulation, gesture recognition, and AR to design reconfigurable bamboo shelters for displaced persons in Myanmar.

Key takeaway

For AI Scientists and Computer Vision Engineers developing human-computer interaction systems, consider integrating AR and gesture/speech recognition to create more intuitive and context-aware applications. Your focus on natural interaction can significantly enhance user experience in fields like retail, construction, and sustainable design, potentially leading to more impactful and user-friendly solutions.

Key insights

Integrating AI, AR, and robotics with natural interaction methods creates innovative human-machine workflows for design and construction.

Principles

• Context-aware AI enhances user experience.
• Natural language interfaces simplify complex tasks.
• Sustainable design benefits from advanced fabrication.

Method

Combine AR for spatial data capture with vision-language AI for contextual recommendations. Integrate gesture recognition and physics simulation for interactive design and robotic fabrication of structures.

In practice

• Use AR for room dimensioning in retail.
• Apply object recognition for automated cost estimation.
• Employ gesture control for mixed reality fabrication.

Topics

• Augmented Reality
• Human-AI Interaction
• Generative AI
• Sustainable Architecture
• Robotic Fabrication

Best for: Computer Vision Engineer, AI Scientist, Research Scientist, AI Engineer, AI Researcher, AI Product Manager

Related on AIssential

• 😺Google is killing the prompt box — AI interfaces are evolving beyond text prompts to context-aware, intent-driven interactions.
• What Happens When AI Understands Space Like You Do? — Generative AI combined with Extended Reality creates "spatial intelligence," enabling natural, adaptive, and scalable immersive experiences.
• AI-Augmented Design Systems: Building Intelligent UX Foundations — AI augments design systems by observing usage and enforcing consistency, but only if systems are structured as measurable data.
• Pixi’s new iOS app turns text messages into interactive AR experiences — Pixi integrates on-device AI with AR to create interactive, context-aware messaging experiences, moving beyond static media.
• When Copying The Physical World Helps The UI, and When It Doesn't — The iPhone keyboard's success stemmed from mimicking physical form while discarding hindering mechanics.
• MIT researchers “speak objects into existence” using AI and robotics — A novel system integrates speech, 3D generative AI, and robotics to fabricate physical objects on demand.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by MIT News - Natural language processing.