AI advice suppresses people's willingness to say "I don't know", even when the advice is wrong and accuracy is incentivized

2026-07-15 · Source: Takara TLDR - Daily AI Papers · Field: Science & Research — Social Sciences & Behavioral Studies, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study involving five experiments (N=3,132) investigated how AI advice influences human judgment, specifically the willingness to admit "I don't know." Participants answered difficult questions, with AI advice intentionally engineered to be wrong. The research found that merely having access to AI, whether actively requested or simply displayed, significantly reduced participants' inclination to suspend judgment. Consequently, while participants answered more questions, their accuracy dropped to about one-third of the baseline without AI, even as their confidence nearly doubled. Although incentivizing accuracy and penalizing errors led to less reliance on AI and improved accuracy, participants still suspended judgment less often than when AI was completely unavailable. This suggests AI suggestions can alter the metacognitive threshold for deciding one's knowledge.

Key takeaway

For AI Scientists designing human-AI interaction systems, this research highlights a critical risk: AI advice can inadvertently erode users' metacognitive awareness. You should prioritize designing AI to explicitly signal uncertainty or encourage "I don't know" responses, especially in domains where accuracy is paramount. Implement mechanisms that incentivize human critical thinking and penalize over-reliance on AI, rather than simply optimizing for response fluency.

Key insights

AI advice suppresses "I don't know" responses, reducing accuracy while boosting confidence, even when wrong.

Principles

• AI access reduces judgment suspension.
• Wrong AI advice boosts user confidence.
• Accuracy incentives mitigate AI over-reliance.

Method

Participants answered difficult questions with an "I don't know" option. AI advice was intentionally wrong. Accuracy was sometimes incentivized.

In practice

• Design AI to encourage uncertainty.
• Incentivize human accuracy over speed.
• Educate users on AI fallibility.

Topics

• Human-AI Interaction
• Metacognition
• Decision Making
• AI Ethics
• Cognitive Bias
• AI Trust

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, Research Scientist, AI Ethicist

Related on AIssential

• The Impossibility of Eliciting Latent Knowledge — The paper proves it's impossible to guarantee AI honesty about latent knowledge using only feedback on agent behavior.
• Tell us: have you used an AI chatbot to make a significant decision – and regretted it? — Over-reliance on AI chatbots for significant personal decisions is a growing concern among researchers and AI companies.
• AI Isn’t Making You Dumber. — AI exposes pre-existing cognitive outsourcing rather than causing new intellectual decline.
• People overestimate how confident AI systems are in their responses, experiments reveal — People overestimate AI confidence, perceiving AI-attributed decisions as more assured than human ones.
• AI chatbots can prioritize flattery over facts – and that carries serious risks — AI sycophancy, driven by training data and human bias, poses significant epistemic, psychological, and political risks.
• The Hidden Risks of AI Nobody Talks About — Over-reliance on AI poses hidden risks to critical thinking, information accuracy, data privacy, and human creativity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.