Building AI product sense, part 2

2026-02-10 · Source: Lenny's Newsletter · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Product Management · Depth: Intermediate, long

Summary

Dr. Marily Nika, an experienced AI Product Manager from Google and Meta, outlines a three-step weekly ritual designed to rapidly build "AI product sense." This skill, crucial for modern product management, involves understanding AI model capabilities and limitations to build user-loved products. The ritual focuses on proactively identifying predictable failure modes before products reach users. It includes mapping failure signatures by asking models to process messy data, testing semantic fragility with ambiguous prompts, and identifying first points of failure with unexpectedly difficult tasks. The process also emphasizes defining a Minimum Viable Quality (MVQ) with acceptable, delight, and do-not-ship thresholds, alongside estimating the feature's cost envelope. Finally, it details designing guardrails to manage model limitations and prevent user confusion or loss of trust.

Key takeaway

For AI Product Managers developing new features, you should implement a weekly ritual to stress-test your AI models. Proactively identifying failure modes, semantic fragility, and performance limits will enable you to design effective guardrails and define clear Minimum Viable Quality (MVQ) thresholds, ensuring your product is robust and trustworthy before it reaches users, thereby preventing costly post-launch issues.

Key insights

Proactive testing of AI model failure modes is crucial for developing robust AI product sense.

Principles

• AI models confidently invent structure from messy inputs.
• Ambiguity is kryptonite for probabilistic AI systems.
• Performance drops when AI features move from dev to production.

Method

Map failure modes by testing with messy, ambiguous, and difficult inputs; define Minimum Viable Quality (MVQ) thresholds; and design guardrails to manage model limitations and ensure trustworthy user experiences.

In practice

• Ask models to extract "strategic decisions" from chaotic Slack threads.
• Summarize a PRD for "execs" to test semantic fragility.
• Estimate model cost per call and usage frequency for cost envelope.

Topics

• AI Product Sense
• Product Management
• AI Model Evaluation
• Minimum Viable Quality
• AI Guardrails

Best for: AI Product Manager, Product Manager, AI Student

Related on AIssential

• Building AI product sense, part 2 — Proactive testing of AI models against real-world chaos builds critical AI product sense and prevents user trust erosion.
• 6 Failure Modes I Test Every AI Startup Idea Against Before Writing Code — Rigorous pre-code validation against known failure modes is vital for AI startup success.
• Why Most AI Features Fail Before the Model Is Even Chosen — Successful AI features prioritize user friction and workflow design over model selection and abstract AI capabilities.
• A Field Guide to Rapidly Improving AI Products — Successful AI product development prioritizes measurement, iteration, and learning over a "tools-first" approach.
• How to test a new AI Model to get up to speed with its capabilities quickly — Developing "Model Sense" is crucial for product builders to create reliable AI-powered products.
• You Trust Your AI Outputs? That’s Cute. Let’s Talk Harness Engineering. — The real engineering challenge is not generating outputs, but controlling AI behavior in production systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Lenny's Newsletter.