San Francisco Demands Apple and Google Delete AI ‘Nudify’ Apps From App Stores

2026-07-17 · Source: WIRED - Ai · Field: Legal & Regulatory — Compliance & Risk Management, Regulatory Affairs & Government Relations · Depth: Fundamental Awareness, medium

Summary

San Francisco City Attorney David Chiu has formally demanded that Apple and Google remove 13 AI "nudify" apps from their App Store and Play Store, respectively. These apps, primarily disguised as face-swapping tools, enable users to create nonconsensual deepfake nude images, overwhelmingly targeting women and girls. Chiu asserts that the tech giants are "aiding and abetting" illegal activity and have likely profited millions from these apps, which use in-app payments. Despite both companies having policies prohibiting such content and Google reporting the suspension of "hundreds" of violating apps, researchers from the Tech Transparency Project and Cornell/Georgetown Universities have repeatedly identified similar "dual-use" apps. These apps, some with over 1 million downloads and collectively estimated to have 480 million downloads and \$120 million in revenue, often lack safety measures, with 70% of tested face-swap apps capable of creating nude images. The city is prepared to consider further legal options if the companies do not comply.

Key takeaway

For legal professionals and policy makers addressing online harm, this situation highlights the urgent need for updated regulations. You should scrutinize app store content policies and enforcement mechanisms, especially concerning "dual-use" AI applications that bypass moderation. Consider advocating for legislation that holds platforms accountable for profiting from illegal content, compelling them to implement proactive screening and robust content removal processes. This proactive stance is crucial to mitigate the severe psychological and reputational damage inflicted by nonconsensual deepfakes.

Key insights

"Dual-use" AI apps evade moderation, enabling nonconsensual deepfake creation despite platform policies and severe harm.

Principles

• Platform policies alone fail to prevent harmful "dual-use" AI apps.
• App stores profit from illicit content via in-app payment cuts.
• Generative AI advancements increase deepfake realism and accessibility.

In practice

• Scrutinize "face-swapping" or generic AI image apps for hidden nudification features.
• Report apps facilitating nonconsensual deepfake creation to platform owners.
• Advocate for stronger platform content moderation and enforcement mechanisms.

Topics

• AI Deepfakes
• Nonconsensual Intimate Images
• App Store Moderation
• Platform Accountability
• Generative AI Ethics
• Digital Harm

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Legal Professional, Policy Maker, Tech Journalist

Related on AIssential

• San Francisco orders Apple, Google to remove nudify apps from app stores — App stores face increasing legal and ethical pressure to remove AI-powered nudification and deepfake content generation apps.
• YouTube and X Have Become ‘Gateways’ to Nudify Apps — Mainstream social media platforms inadvertently enable the proliferation of nonconsensual deepfake tools despite explicit policies.
• Musk’s tactic of blaming users for Grok sex images may be foiled by EU law — The EU is banning AI "nudify" apps, holding developers accountable for harmful outputs, not just users.
• New Study Examines Features and Policies for 29 AI ‘Undressing’ Apps — AI "undressing" apps are vehicles for systemic gender-based violence, objectifying women and commodifying their exploitation.
• The problem AI content moderation cannot solve — AI content moderation fails to address image-based abuse because it cannot account for consent and context, focusing only on explicit content.
• Copyright and AI Policy Needs Precision, Not Panic — AI policy needs nuanced solutions that protect creators without empowering corporate intermediaries or stifling innovation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by WIRED - Ai.