When A Chatbot Becomes Your Therapist

2026-03-12 · Source: Partnership on AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Ethics & Societal Impact · Depth: Intermediate, medium

Summary

Millions of people are using AI chatbots, including general-purpose tools like ChatGPT and Claude, for mental health support, often sharing vulnerable information such as struggles with loneliness or suicidal thoughts. This widespread use occurs largely without transparent safety frameworks or clinical validation, despite allegations that some chatbots underestimate suicide risk and may contribute to user suicides. The global mental health crisis, marked by over one in five U.S. adults living with mental illness and significant barriers to care, drives many to seek AI assistance. The Partnership on AI (PAI) identifies three systemic challenges: AI development outpacing mental health research, companies working in isolation, and a lack of independent evaluations. To address these, PAI recently hosted a workshop at OpenAI with companies like Anthropic, Meta, and OpenAI, alongside mental health institutions, to develop best practices for handling high-stakes mental health interactions, specifically focusing on suicide prevention and non-suicidal self-injury (NSSI).

Key takeaway

For CTOs and VPs of Engineering overseeing AI product development, your teams must prioritize integrating clinical expertise and independent evaluation into AI mental health applications. Actively participate in multi-stakeholder collaborations, like those facilitated by PAI, to share insights and align on best practices for high-stakes interactions such as suicide prevention. This proactive engagement is crucial to mitigate risks, build trust, and ensure your AI tools provide meaningful support rather than compounding harm, especially given the rapid pace of AI advancement and current reliance on these systems.

Key insights

AI chatbots are widely used for mental health support, necessitating urgent, collaborative safety frameworks and independent evaluation.

Principles

• AI development outpaces mental health research.
• Collaboration is vital for integrating clinical expertise into AI product decisions.
• Independent evaluation is crucial for AI mental health tools.

Method

PAI convened frontier AI companies and mental health experts in a workshop to align on current practices, identify convergence/divergence, and establish 3-5 high-priority normative best practices for suicide prevention and non-suicidal self-injury.

In practice

• Share practical lessons on emerging patterns and tested interventions.
• Establish an ongoing AI and Mental Health advisory group.
• Develop detailed guidance for implementing and evaluating best practices.

Topics

• AI Chatbots
• Mental Health Support
• AI Ethics
• Suicide Prevention
• Responsible AI

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Ethicist, Policy Maker, AI Product Manager

Related on AIssential

• ChatGPT as a therapist? New study reveals serious ethical risks — AI chatbots, even when prompted as therapists, consistently fail to meet professional mental health ethical standards.
• We Need to Know More About How AI is Affecting Mental Health — A critical knowledge gap exists regarding AI's mental health impacts, demanding urgent research and regulation amidst widespread chatbot adoption.
• Just how bad are generative AI chatbots for our mental health? — Media overemphasizes severe AI chatbot-related mental health harms, often misattributing causality due to "compassion illusions."
• Chatbots Need Guardrails to Prevent Delusions and Psychosis — AI companions pose mental health risks, necessitating robust guardrails, independent audits, and legislative oversight to prevent harm.
• Unregulated chatbots are putting lives at risk | Letters — Unregulated AI chatbots pose significant mental health risks due to a lack of pre-use screening for vulnerable users.
• I Built an AI Mental Health App at 16. Here’s What It Did to My Own Mental Health. — Building AI for mental health can paradoxically hinder personal emotional processing, despite the AI's effectiveness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Partnership on AI.