How to actually protect against digital sexualized violence

Generative AI is increasingly being used as a tool for sexualized violence against women and girls. Without their consent, men create sexualized deepfakes, generate undressed bodies using AI tools, which they share on social media platforms such as X (formerly Twitter) and spread sexist insults via AI chatbots. What we see in the digital sphere is not an isolated problem but a continuation of patriarchal power structures and gender-specific violence through new means.

The consequences for women that are affected are severe: they could suffer from significant psychological and physical distress but only have a few options to defend themselves. In addition, sexualized deepfakes are used to intimidate women in order to push them out of public and political debates.

The regulatory landscape currently consists of different European and national rules that are not well coordinated and, despite their diversity, still leave gaps that need to be closed. In Germany, there is currently a lack of reliable data on gender-based violence in the digital space, as well as a lack of a legal framework that adequately addresses this issue.

At the EU-level, there are discussions on how to better protect women as part of the overhaul of the AI Act. The Digital Services Act (DSA) primarily addresses how such content is disseminated on platforms.

Ban AI that can be used to create sexual deepfakes

Today, AI-based tools make it easier than ever to create non-consensual sexualized deepfakes. These range from applications that specifically advertise their ability to sexualize images of real people to general-purpose software such as face-swapping apps, which can be misused for such purposes.

The European AI Act establishes rules for the providers and deployers of AI systems and is currently being overhauled as part of the so-called AI Omnibus. In our view, it is problematic that within this process, an attempt is made to overturn important safety guards. A ban on AI systems that allow the creation of non-consensual sexualized deepfakes is one of the few positive proposals that are negotiated within the Omnibus. Such a ban would close a long-overdue gap in the law and would be a significant step to better protect those affected. Because the addition to the AI Act would specifically hold AI companies accountable.

Our demands for a ban on non-consensual sexualized deepfakes

For such a ban to be effective and proportionate, a difficult balance must be found. Consent must be clearly defined, liability must be assigned unambiguously, and safeguards must be appropriate so that legitimate use and open-source development of AI systems are not hindered.

It is crucial that not all AI systems that can be used to create images fall under general suspicion. The issue concerns services that explicitly make it easier to create problematic content, fail to implement adequate safety measures, and offer no option to prevent misuse. But even systems with safeguards are not immune to attempts to circumvent them. Where third parties deliberately remove existing security measures, they must bear the liability. We have summarized our detailed perspective on the matter in a position paper:

• Precisely define consent: free, informed, context-specific, and explicit; no application in the context of content of non-real people.
• Clearly assign liability: Only collect data from AI systems that do not have adequate safeguards.
• Avoid unintended side effects: Do not jeopardize open-source AI development through overregulation.
• Strengthen security measures: Implement adequate safeguards with continuous monitoring and reporting mechanisms.
• Consent requests: Make them mandatory for depictions of real people.

Platforms must finally be held accountable as well: No more half-hearted enforcement

When algorithms of platforms are designed to amplify content that creates a lot of engagement and has an emotional or scandalizing effect, sexualized deepfakes are shown to even more people, causing further harm to women affected. Big platforms are also used to share links to problematic apps, advertise them, and discuss tips on how to turn typical, consumer-oriented LLMs into so-called “non-consensual sexualization tools” (NSTs for short, also known as “nudify apps”).

At the EU-level, die Digital Services Act (DSA) is already in effect, requiring very large online platforms (with more than 45 million active users per month within the EU) to mitigate so-called “systemic risks”. This includes gender-based violence. However, enforcing this requirement in practice is often difficult. For instance, X (formerly Twitter) consistently refused to grand AlgorithmWatch access to data, even though the DSA mandates the release of said data for research purposes.

This way, X is actively preventing us from investigating these risks, even though the company is required by law to mitigate them. The European Commission must prevent and punish such behavior faster and more consistently in combination with substantial fines for platforms that do not meet their obligations. Furthermore, it should be examined whether people affected are entitled to compensation if they suffer harm from such content being shared and not removed fast enough.

In member states where non-consensual sexualization is already illegal, all online services (not only the very large ones) have to take action against illegal content on their platforms and report it. In the case of Germany, non-consensual sexualizing deepfakes do not constitute a criminal offence on their own yet; however, a corresponding draft bill against digital violence is currently being developed by the Federal Ministry of Justice.

This is an important and long-overdue step. However, the initiative could fall short of its goal if it is not supplemented by clear and enforceable obligations for platforms and AI companies. Furthermore, the proposed changes are designed as offenses requiring a complaint to be filed, placing the burden on those affected.

Punish perpetrators, identify systemic issues, recognize protection as a societal responsibility

This makes it even more essential to promote not only legal measures but also societal changes in how we address gender-based violence and support those affected. Women affected by sexualized deepfakes are not to blame – the responsibility lies with the perpetrators, platforms, and AI companies. Nonetheless, the women affected often experience stigmatization, shame, and damage to their reputation, which can keep them from seeking help or filing a report. We need safe points of contact where victims are believed, low-barrier access to therapeutic support is provided – also during ongoing legal proceedings – and a societal awareness that non-consensual sexualized deepfakes are a crime for which not the victims have to justify their actions, but the perpetrators.