California is strengthening its regulatory grip on artificial intelligence with the launch of a new California AI oversight unit, as the state continues investigating Elon Musk’s xAI over the generation of non-consensual explicit content.

California Attorney General Rob Bonta confirmed that his office is building an artificial intelligence accountability program while probing xAI’s chatbot Grok for producing sexually explicit images involving adults and potentially minors. The move signals California’s expanding role as a global watchdog for AI safety and regulation.

The attorney general’s office issued a cease-and-desist letter to xAI last month, joining regulators worldwide who are scrutinizing the company’s AI-generated content practices.

xAI Investigation and Regulatory Concerns

According to Bonta, the California AI oversight unit is seeking confirmation that the problematic conduct has stopped and remains in discussions with xAI. He said the company has deflected responsibility while still allowing certain forms of sexualized content generation for paying users.

“Just because you stop going forward doesn’t mean you get a pass on what you did,” Bonta said.

xAI, recently acquired by SpaceX, did not respond to requests for comment. Earlier this year, the company stated that it had introduced safeguards to reject requests for sexualized images of real individuals and block such content in regions where it violates local laws.

Despite these measures, regulators continue to assess whether the company’s safeguards are sufficient.

California Strengthens Its Role as an AI Watchdog

The formation of the California AI oversight unit highlights the state’s growing commitment to AI accountability, even as some industry leaders and lawmakers push for federal authorities to lead regulation efforts.

Bonta warned against granting Congress exclusive control over AI regulation, pointing to past legislative gridlock on data protection and emerging technologies. Instead, California is increasing its internal expertise through an “AI oversight, accountability and regulation program.”

The initiative aims to address risks posed by AI systems, particularly those involving harmful chatbot interactions. Bonta emphasized that AI chatbots engaging in explicit conversations with minors or encouraging self-harm are unacceptable.

Broader AI Safety Efforts and Legislative Action

California’s regulatory push extends beyond xAI. State authorities have also informed San Francisco-based OpenAI that California maintains an ongoing interest in ensuring the safety of its products and services, particularly following the company’s corporate restructuring last year.

Meanwhile, California lawmakers are considering legislation that would formally require the attorney general’s office to build AI expertise and strengthen regulatory oversight capabilities.

In a joint interview, Connecticut Attorney General William Tong described AI-related harm as a major consumer protection challenge, comparing its significance to past public health crises. He warned that the impact of AI and social media risks could affect future generations.

Growing Momentum for AI Regulation

The creation of the California AI oversight unit reflects mounting global concern over AI safety, accountability and consumer protection. As governments grapple with the rapid expansion of AI technologies, California’s approach may shape future regulatory frameworks worldwide.