Meta secretly tested ChatGPT, Gemini, and Character.AI with thousands of minor-perspective crisis prompts
Summary
Meta secretly conducted extensive safety testing on rival AI chatbots, including ChatGPT, Gemini, and Character.AI, using thousands of crisis prompts from the perspective of minors. Internally named "Cannes," the project involved hundreds of contractors from Covalen creating fake underage accounts to send prompts about self-harm, eating disorders, and drugs. Active through April 2026, one August 2025 round saw over 45,000 prompts. Meta defended this as industry-standard safety testing, stating the collected responses were not used for training its own AI models. However, the tested companies, unaware of the activity, indicated potential terms of service violations, with Character.AI confirming a breach.
Key takeaway
For AI developers and product managers designing or deploying conversational AI, you must prioritize transparent and ethical safety testing protocols, especially when potential interactions involve minors. Ensure your terms of service explicitly address such testing scenarios and consider robust age verification mechanisms to mitigate risks. This incident underscores the critical need for industry-wide standards to prevent unauthorized data collection and potential legal or reputational damage.
Key insights
Covertly testing AI chatbots with minor-perspective crisis prompts raises significant ethical, legal, and terms of service concerns.
Principles
- AI safety testing requires transparency and consent.
- Age verification for AI chatbots is largely absent.
- Violating terms of service can have repercussions.
Topics
- AI Safety
- Chatbot Ethics
- Minor Protection
- Generative AI
- AI Testing
Best for: CTO, Executive, VP of Engineering/Data, Tech Journalist, AI Ethicist, Legal Professional
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by The Decoder.