From Risk Mitigation to App Bans: Assessing EU Legislation’s Potential to Combat AI-generated Image Abuse

In late December 2025, X rolled out a new AI-based picture-editing feature in its interface. Users could now edit pictures posted to the website by simply addressing X’s AI-chatbot Grok in a response to the posted picture. What ensued was an avalanche of non-consensual sexualised deepfakes of women and girls, created and shared directly on X. The deepfakes targeted women, particularly public figures, as well as minors, which rightly garnered wide-reaching public outrage, prompting an investigation by the EU Commission into X’s compliance with key provisions of the DSA.

However, the Grok scandal brought to light a much bigger issue: the proliferation of nudification tools which are predominantly being used to create non-consensual sexualised deepfakes of women and minors. A recent investigation by the Tech Transparency Project identified 55 nudification apps in the Google Play Store and 47 in the Apple App Store. In our new brief, CDT Europe assesses the potential of EU legislative frameworks to address this issue.

The AI Act’s risk assessment and mitigation obligations for certain general-purpose AI model providers can offer protections against the generation and dissemination of AI-generated NCII and CSAM. In light of the law’s limitations, voices calling for additional safeguards and protections against the AI-facilitated generation and dissemination of NCII and CSAM under EU law have however grown stronger. Several political groups have therefore proposed to include a prohibition of such practices under the AI Act in the context of the negotiations on the AI Omnibus. Keeping in mind constraints related to the scope, feasibility and effectiveness of such a measure, we argue that at a minimum, a ban could include “nudification apps”, i.e. AI systems specifically intended to be used for the generation of sexually explicit imagery and/or CSAM.

Beyond the AI Act, we find that the regulatory landscape in its current state, which also includes the DSA, the Directive and the GDPR, offers clear opportunities to limit the prevalence of AI-generated NCII and CSAM online, despite certain limitations. Due to the multifaceted nature of this content, its generation and its proliferation, a multi-pronged approach is key. Therefore, in this brief, we argue it is crucial that existing provisions as well as upcoming legislation are enforced consistently, swiftly and transparently both by the Commission and, where relevant, by member states. Only a strong and consistent enforcement and implementation of these tools will allow for them to reach their potential and thus mitigate harm done and prevent future harm, protecting women, girls and other vulnerable groups on and offline and making the internet safe for all.

Read the full brief here.