Sexualized violence in the digital sphere: Who is affected, what are the consequences, and how can you defend yourself?

2023-09-18 · Source: AlgorithmWatch · Field: Government & Public Sector — Public Policy & Governance, Public Safety & Security, Regulatory & Compliance · Depth: Intermediate, extended

Summary

Sexualized violence in the digital sphere, particularly non-consensual deepfakes, is escalating, disproportionately affecting women, children, and the LGBTQI+ community. Generative AI tools like "Nudify" apps and "Face Swaps" facilitate the creation of manipulated nude images from everyday photos without consent. High-profile cases include Kamala Harris and Collien Fernandes, while the X AI Chatbot Grok generated over 7,800 sexual deepfakes in January 2026. Studies reveal a 550% increase in deepfake videos since 2019, with 98% being sexual and 99% depicting women. In Germany, 18.3% of women experienced digital violence in the past five years, and a 2025 study found 63% of politically active women faced such violence. Legal frameworks like the EU Digital Services Act, the European AI Act, and Germany's proposed Digital Violence Protection Act aim to address this, but enforcement gaps and societal victim-blaming persist, hindering effective protection and support for victims.

Key takeaway

For policy makers developing digital safety legislation, you must prioritize comprehensive laws that explicitly criminalize the creation and distribution of non-consensual sexualized deepfakes. Ensure these laws, like the EU AI Act and national initiatives, are consistently enforced by adequately trained law enforcement and judicial bodies. Your efforts should also foster a societal shift to combat victim-blaming and normalize reporting, thereby protecting vulnerable groups and preserving democratic participation.

Key insights

Generative AI fuels a surge in non-consensual sexualized deepfakes, extending patriarchal violence and silencing marginalized groups online.

Principles

• Digital sexualized violence extends analog patriarchal power dynamics.
• AI-generated deepfakes disproportionately target women, children, and LGBTQI+ individuals.
• Effective protection requires consistent legal enforcement and cultural shifts.

In practice

• Recognize deepfakes as gender-based violence, not just tech abuse.
• Advocate for robust legal enforcement and AI regulation.
• Challenge victim-blaming and societal normalization of digital violence.

Topics

• Non-consensual Deepfakes
• Generative AI Misuse
• Digital Gender Violence
• AI Act
• Digital Services Act
• Victim Protection

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Policy Maker, AI Ethicist, Legal Professional

Related on AIssential

• How the AI Deepfake Boom is Outpacing Europe's Safeguards — AI-powered "nudification" tools enable widespread, realistic sexualized deepfakes, exacerbating digital violence and outpacing regulatory and institutional responses.
• How to actually protect against digital sexualized violence — Effective protection against digital sexualized violence demands accountability from AI companies, platforms, and perpetrators via robust legislation.
• Click, Strip, Repeat: Sex Workers and Digital Violence Amidst the Deepfake Boom — Non-consensual sexualization tools (NSTs) weaponize AI-generated deepfakes, causing profound harm, especially to women and sex workers, exacerbated by systemic misogyny.
• New Study Examines Features and Policies for 29 AI ‘Undressing’ Apps — AI "undressing" apps are vehicles for systemic gender-based violence, objectifying women and commodifying their exploitation.
• Grok’s Deepfake Lawsuits: courts, regulators, and plaintiffs’ firms are no longer treating sexually exploitative deepfakes as mere “misuse by bad actors.” They are increasingly framing them as... — AI developers face increasing legal and regulatory pressure to treat sexually exploitative deepfakes as foreseeable product outcomes.
• Amount of AI-generated child sexual abuse material found online surged in 2025 — AI advancements are significantly increasing the volume and severity of online child sexual abuse material.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AlgorithmWatch.