The online platform X has sparked widespread backlash after users employed its artificial intelligence assistant Grok to manipulate photos of women by taking off their garments.
Several cases have been seen where the system has been used to digitally undress female subjects to make them appear in swimwear without permission, as well as to put them in inappropriate situations.
Company Response and Legal Position
XAI, the organization that created Grok, did not respond to requests for comment, aside from an automatically-generated message stating "traditional news outlets lies".
A Home Office spokesperson stated that new laws is being introduced to outlaw nudification technology. Under a new law, anyone who distributed such tools would "be subject to a jail term and heavy penalties".
"X or Grok could prevent these forms of misuse if they desired to," said a law professor, adding they "operate with a lack of consequences."
Regulator Stance and Platform Functionality
The regulator Ofcom said tech firms must "evaluate the risk" of people viewing illegal content on their networks, but did not confirm whether it was currently investigating X or Grok in regarding AI-generated images.
Grok is a no-cost chatbot—with some premium features—which reacts to X users' requests when they tag it in a post.
It is commonly used to provide commentary or additional background to other posters' remarks, but people on X are also can edit an posted picture through its AI image editing tool.
Background of Criticism and Rules
The system has been condemned for allowing users to produce pictures and footage with nudity and sexually suggestive content, and it was previously alleged to have been involved in creating graphic content of public figures.
"The platform has been allowing the production and spread of these images for a long time without taking any action and we have not seen any intervention by oversight bodies," Professor McGlynn added.
XAI's own terms of service bans "showing images of individuals in a pornographic manner".
In a statement, Ofcom said it was against the law to "generate or distribute private pictures or child sexual abuse material" and confirmed this included explicit AI images generated with AI.
It said services such as X were required to take "suitable measures" to "minimize the danger" of users viewing illegal content on their platforms and to remove it swiftly when they are notified of it.
