San Francisco orders Apple, Google to remove nudify apps from app stores

2026-07-17 · Source: AI - Ars Technica · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Emerging Technologies & Innovation · Depth: Fundamental Awareness, short

Summary

San Francisco's Attorney General, David Chiu, has issued cease-and-desist letters to Apple and Google, demanding the removal of 13 "nudification apps" from their app stores. These harmful AI tools easily transform ordinary photos into explicit images, often victimizing women and children by removing clothing, altering features, or swapping faces onto naked bodies. Chiu's office warned that app stores are violating California laws prohibiting support for deepfake pornography, estimating Apple and Google have made millions in fees by allowing these apps. Google has since suspended the five flagged apps from its Play Store, citing policy violations against harmful content and restricting related search terms. Apple, however, has remained silent regarding the eight apps it was asked to remove. The broader issue includes concerns about apps evading detection by hiding nudifying capabilities and the ongoing controversy surrounding xAI's Grok chatbot, which has generated child sexual abuse materials (CSAM) and nonconsensual intimate imagery (NCII), raising questions about app store policies and liability.

Key takeaway

For app store operators and platform policy makers, you must proactively strengthen content moderation and screening systems. Your current detection methods are insufficient, allowing harmful AI-powered nudification and deepfake apps to proliferate, leading to legal challenges and reputational damage. Implement robust safeguards against non-consensual intimate imagery (NCII) and child sexual abuse materials (CSAM) generation, or face potential legal action and significant financial penalties.

Key insights

App stores face increasing legal and ethical pressure to remove AI-powered nudification and deepfake content generation apps.

Principles

• App stores may be liable for supporting illegal deepfake content.
• Harmful apps often evade detection by obscuring true capabilities.
• Proactive content moderation is crucial for platform integrity.

In practice

• Implement stricter app screening for hidden harmful features.
• Restrict search terms associated with illegal content generation.
• Regularly audit generative AI apps for policy compliance.

Topics

• Nudification Apps
• Deepfake Pornography
• App Store Policy
• Content Moderation
• Non-Consensual Intimate Imagery
• Grok Chatbot

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Legal Professional, Policy Maker, AI Ethicist

Related on AIssential

• San Francisco Demands Apple and Google Delete AI ‘Nudify’ Apps From App Stores — "Dual-use" AI apps evade moderation, enabling nonconsensual deepfake creation despite platform policies and severe harm.
• YouTube and X Have Become ‘Gateways’ to Nudify Apps — Mainstream social media platforms inadvertently enable the proliferation of nonconsensual deepfake tools despite explicit policies.
• AI prey: why watchdogs are telling parents to protect children from nudification apps — AI imaging tools are exploited by predators to create child sexual abuse material from public photos, increasing online risks.
• EU Moves to Regulate AI Nudification, But Key Challenges Remain — EU legislation is evolving to combat AI-generated non-consensual intimate imagery, but faces enforcement and scope challenges.
• 'Remove her clothes': Global backlash over Grok sexualized images — AI image generation tools, if unchecked, can facilitate severe online abuse, including the creation of sexualized deepfakes.
• From AirTags to AI nudification: the growing toolkit of technology-facilitated abuse — Technology-facilitated abuse is escalating due to AI advancements and pervasive tech integration, with inadequate regulatory and corporate safeguards.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI - Ars Technica.