xAI has initiated a lawsuit against Minnesota Attorney General Keith Ellison, challenging a state law enacted in May that broadly addresses “nudification” applications. The company contends that the statute's stringent provisions compel it to “restrict Grok Imagine’s image-editing features in various ways,” thereby infringing upon its First Amendment rights.
This legal challenge follows an incident in January when Grok inundated the internet with millions of sexually explicit deepfakes, some depicting minors. Historically, a content moderation failure of this scale would typically prompt a swift shutdown of the product. However, under Elon Musk's leadership, the chatbot continued to generate such images for days, with Musk notably posting, “Grok now hitting #1 on the App Store in one country after another!” amidst the deepfake proliferation.
The incident highlighted a significant flaw, raising questions globally about the adequacy of existing laws to curb the surge of nonconsensual nude images. Governments worldwide reacted, with both the European Union and the United Kingdom launching investigations, and officials in France, India, and Malaysia expressing strong disapproval. While the U.S. federal government maintained a more subdued stance, states like California and Minnesota adopted stricter measures.
Minnesota's state legislature subsequently passed a law specifically targeting “nudification” technology. This statute prohibits site owners from enabling individuals to “access, download, or use” such software, though it includes an exception for technologies requiring “the technical skill of a user to nudify an image or video,” thereby excluding tools like Photoshop from its scope. The legislation also establishes a civil cause of action for private citizens and empowers the state attorney general to levy penalties up to $500,000 per violation.
This pioneering law, signed by Governor Tim Walz in May, was slated to become effective on August 1st. A curious aspect of the situation is xAI's delayed legal action; rather than filing in the preceding three months, the company initiated its lawsuit just days before the statute's implementation. This timing suggests a potential last-minute realization within xAI of impending violation of a highly punitive law, prompting a hurried search for legal counsel willing to work intensively on behalf of a platform that had generated nonconsensual deepfakes. One might surmise the legal professionals involved are commanding substantial fees.
The lawsuit's complaint, upon examination, presents what could be considered reasonable arguments. It asserts that the Minnesota law is excessively broad, imposing liability even on consensually generated images, adult images deemed “sort-of kind-of sexy,” or those possessing “humorous” or “artistic” merit. The First Amendment, the complaint emphasizes, mandates that laws be precisely drafted to avoid prohibiting forms of social and political commentary, citing an example like a deepfake depicting Chris Christie and JB Pritzker sumo wrestling.
The lawsuit earnestly declares that “Liability also attaches … even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images.” It further laments the strict liability imposed for unauthorized user actions, specifically for “images and videos that users might manage to create in violation of Grok’s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.”
To underscore its commitment, the lawsuit asserts, “xAI takes violations of its terms of service seriously,” highlighting that the company has subsequently sued one of its users for generating CSAM (Child Sexual Abuse Material) using Grok. Notably, this lawsuit against the alleged CSAM creator was initiated more than two months after the Minnesota statute was enacted.
While xAI’s legal team may present valid points, their lawsuit's tone has been characterized as somewhat detached from the immediate context. They contend that existing Minnesota and federal legislation, such as the TAKE IT DOWN Act, already effectively prohibits the nonconsensual dissemination of nude deepfakes. These established statutes, the lawyers argue, are constitutionally sound, a claim they do not extend to Minnesota's new “nudification” law. Furthermore, they emphasize that any American law targeting speech based on its content must be drafted with the least possible restriction while still serving a “compelling” government interest.
However, this argument brings into focus the core issue highlighted by the “Grok deepfake-ocalypse.” If existing legal frameworks were truly adequate, January would not have seen such a deluge of nonconsensual pornographic content. A report by the Center for Countering Digital Hate, analyzing an 11-day period from December 29th to January 8th, revealed that Grok produced approximately 3 million sexualized images, with about 23,000 of these depicting children. The report starkly concluded, “That is a shocking rate of one sexualized image of a child every 41 seconds.”
It is important to note that many of these generated images might not, in fact, be explicitly illegal under current statutes. For example, the legal status of a suggestive deepfake of a minor in a bikini as CSAM is not definitively clear, though one expert previously informed The Verge that other images Grok produced, such as those depicting minors’ faces covered in semen, likely did contravene existing law.
Regardless of the specific legality of each image, the fact that a multibillion-dollar company permitted Grok to operate unchecked for such an extended period strongly indicates that existing legal frameworks failed to sufficiently incentivize responsible behavior from the involved parties. While Minnesota's HF 1606 might be criticized as an imperfect law, this raises a more fundamental question: what recourse is left when even the “good” laws prove ineffective?
The Editorial Staff at AIChief is a team of professional content writers with extensive experience in AI and marketing. Founded in 2025, AIChief has quickly grown into the largest free AI resource hub in the industry.