How to do AI analysis you can actually trust

2026-02-10 · Source: Lenny's Newsletter · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, long

Summary

Caitlin Sullivan, an expert in AI-powered user research, identifies four common failure modes when using large language models (LLMs) like ChatGPT, Claude, and Gemini for analyzing customer data from interviews and surveys. These issues include invented evidence (hallucinations or "Frankenstein quotes"), false or generic insights, signal that doesn't guide decisions, and contradictory insights. Sullivan emphasizes that AI outputs often appear confident even when flawed, leading to misguided product decisions. She highlights that LLMs struggle with unstructured interview data by oversimplifying and with survey data by misinterpreting sparse responses or irrelevant metadata. The article also compares LLMs, recommending Claude for thorough analysis, Gemini for evidenced themes, and ChatGPT for final framing, noting ChatGPT's higher propensity for the discussed failure modes.

Key takeaway

For Product Managers and Research Scientists analyzing customer feedback with AI, you must implement rigorous prompting and verification steps to avoid critical errors. Define precise quote rules and use a secondary verification prompt to confirm the authenticity of AI-generated evidence. This prevents acting on invented or generic insights, ensuring your product decisions are grounded in actual customer voice rather than confident AI hallucinations.

Key insights

AI analysis of customer data often yields confident but flawed insights, requiring specific prompting techniques for reliability.

Principles

• AI defaults to finding consensus, losing valuable edge cases.
• LLMs generate text, not retrieve quotes like a search engine.
• Verification is crucial to prevent AI-driven confirmation bias.

Method

Define explicit quote rules (start/end, content, citation) and then use a verification prompt to confirm quote existence and accuracy against source transcripts, flagging paraphrases or unlocated text.

In practice

• Use Claude for in-depth customer data analysis.
• Define clear project, business, and product context for LLMs.
• Verify all AI-generated quotes against original source material.

Topics

• AI User Research
• Prompt Engineering
• LLM Analysis
• AI Hallucinations
• Qualitative Data Analysis

Best for: Product Manager, Research Scientist, AI Data Scientist

Related on AIssential

• How to do AI analysis you can actually trust — AI-generated customer insights require specific prompting and verification to overcome inherent biases and prevent critical decision errors.
• The AI Answer You Trusted Most Was Probably the Wrong One — AI models are more confident when incorrect, a byproduct of training that rewards human preference for decisive answers.
• stop lying to your ai — AI requires brutally honest, explicit communication to deliver desired outputs, unlike human interaction.
• i tested basically every AI tool i could find for med school research. most are useless unless you already know enough to catch them lying. — AI tools for professional research require extensive manual verification due to persistent accuracy and citation issues.
• Make AI defend its technical decisions — Challenging LLMs to defend their arguments elicits more robust, evidence-based responses.
• AI keeps getting smarter, so why does it still fail at obvious things? — AI's apparent intelligence stems from pattern matching, not true comprehension, leading to reliability gaps.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Lenny's Newsletter.