Building AI product sense, part 2

2026-01-27 · Source: Lenny's Newsletter · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Project & Product Management · Depth: Intermediate, long

Summary

Dr. Marilee Nika, an experienced AI Product Manager from Google and Meta, introduces a weekly ritual designed to rapidly build "AI product sense" for product managers. This skill is crucial for navigating the uncertainties of AI product development, especially when models encounter messy real-world user inputs. Meta's new "Product Sense with AI" interview reflects this shift, evaluating candidates on handling uncertainty and making clear decisions despite imperfect information, rather than prompt engineering or model trivia. The ritual involves three steps: mapping failure modes and intended behavior, defining Minimum Viable Quality (MVQ), and designing guardrails. Nika emphasizes that AI often breaks in production due to predictable failure modes, highlighting the need to proactively identify and design around these issues before user trust erodes. The process aims to clarify whether issues are product problems or model limitations, ultimately helping AI products survive contact with reality.

Key takeaway

For AI Product Managers building new features, proactively integrating a weekly ritual to test model behavior against messy, ambiguous, and difficult real-world scenarios is crucial. This practice will help you identify failure modes and semantic fragility early, enabling you to define clear Minimum Viable Quality (MVQ) thresholds and design effective guardrails. By doing so, you can ensure your AI products are robust and trustworthy before they reach users, preventing costly post-launch issues and preserving user confidence.

Key insights

Proactive testing of AI models against real-world chaos builds critical AI product sense and prevents user trust erosion.

Principles

• AI product sense is the new core PM skill.
• Models confidently invent structure when confronted with mess.
• Ambiguity is kryptonite for probabilistic systems.

Method

The ritual involves three steps: 1) Map failure modes by testing models with obviously wrong, ambiguous, and unexpectedly difficult inputs. 2) Define Minimum Viable Quality (MVQ) with acceptable, delight, and do-not-ship thresholds, considering cost. 3) Design guardrails to protect users when models hit limits.

In practice

• Ask models to extract decisions from chaotic data (e.g., Slack threads).
• Test model semantic fragility with ambiguous prompts (e.g., "summarize this PRD for execs").
• Estimate cost envelope early: model cost per call, usage frequency, caching potential.

Topics

• AI Product Sense
• Model Failure Modes
• Generative AI
• AI Guardrails
• Minimum Viable Quality

Best for: AI Product Manager, Product Manager, AI Student

Related on AIssential

• Building AI product sense, part 2 — Proactive testing of AI model failure modes is crucial for developing robust AI product sense.
• If You Can’t Explain These 5 AI Terms, You’re Just Vibe Coding (A Detailed Guide to AI Terms) — Understanding core AI terms and their failure modes is crucial for building reliable AI systems.
• Why AI Systems Struggle With Truth, Trust, and Reliability — AI systems prioritize believable language over verified truth, creating significant trust and reliability challenges.
• Why Your AI Can Be Brilliant and Still Make Dumb Mistakes — AI failures are predictable, stemming from its nature as a pattern-matching prediction system, not a reasoning agent.
• Building Trustworthy AI for Enterprise Workflows - with Amar Akshat of PaySafe — Enterprise AI requires embedded guardrails and auditable agentic decisions to bridge the consistency gap between demos and production.
• 6 Failure Modes I Test Every AI Startup Idea Against Before Writing Code — Rigorous pre-code validation against known failure modes is vital for AI startup success.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Lenny's Newsletter.