The Perfume Problem: Why Most AI Demos Smell Amazing and Disappear Just as Fast

2026-01-05 · Source: Modern Data 101 · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Emerging Technologies & Innovation · Depth: Intermediate, medium

Summary

Salil Athalye, Senior Director of Solutions Engineering at The Modern Data Company, discusses the deceptive nature of "arresting" AI demos, likening them to fleeting fragrances designed to captivate before skepticism arises. He explains that these demos, often seen at tech conferences, are meticulously choreographed with pre-selected queries, cleaned data, and tuned response times to create an illusion of seamless intelligence, particularly in conversational interfaces layered on BI platforms. Athalye emphasizes that while impressive, these demonstrations often mask the underlying complexities and costs associated with data preparation, semantic model maintenance, and handling unanticipated questions. He argues that the "evanescence" of these demos is intentional, driving purchasing decisions by making the incredible seem briefly attainable, only to disappear when confronted with real-world data and infrastructure.

Key takeaway

For CTOs evaluating new AI-powered BI solutions, you must look beyond the initial "arresting" demo. Focus your due diligence on understanding the actual data plumbing, how the system handles unexpected queries, and its performance in a real-world, long-term operational context. Insist on seeing the underlying architecture and speaking with reference customers to avoid investing in a solution that delivers only a fleeting impression rather than sustained value.

Key insights

AI demos are often meticulously choreographed to impress, masking underlying complexities and potential real-world limitations.

Principles

• Demos prioritize sensory impact over functional depth.
• Evanescence drives desire and purchasing decisions.
• Substance behind the demo is rarer than implied.

Method

To evaluate AI demos, inquire about data preparation, observe failure modes with unscripted questions, and seek reference customers for day-90 "live" performance.

In practice

• Ask vendors about data preparation processes.
• Test demos with unscripted, challenging questions.
• Request reference customers for long-term usage insights.

Topics

• AI Demonstrations
• Data Preparation
• Natural Language Interfaces
• AI System Evaluation
• Implementation Challenges

Best for: CTO, Director of AI/ML, VP of Engineering/Data, AI Product Manager

Related on AIssential

• The AI Promise: A Pyramid View of What Lingers — AI initiatives require robust "heart" (platform) and "base" (people/governance) layers to deliver lasting value beyond initial "top note" demos.
• Mining Architectural Quality Under Agentic AI Adoption: A Causal Study of Java Repositories — Agentic AI tools expand code volume in Java projects without proportionally increasing architectural smells.
• How Braintrust uses AI agents, evals, and CI to ship better software | Ankur Goyal — AI agents and rigorous evals encode "taste" and success criteria, enabling automated, high-quality software development.
• 🔮 Exponential View #567: How AI is rewiring work — The AI economy is rapidly shifting from training to inference, driven by agentic systems and requiring new compute architectures.
• AI enthusiasts are in a race against time, AI skeptics are in a race against entropy — The core tension between AI enthusiasm and skepticism stems from a lack of feedback loops, posing existential threats to teams.
• AI Dev 26 x SF | Ara Khan: Evals Are Broken Use Them Anyway — Evals are imperfect but essential for AI agent development, requiring nuanced interpretation and iterative improvement.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Modern Data 101.