A rogue OpenAI AI model hacked HuggingFace on its own; the company used a Chinese AI to contain it

For months, we have heard that models like Anthropic’s Claude Mythos and OpenAI’s GPT-5.6 are the most advanced in existence. These models have the potential to transform cybersecurity as we know it. However, in a recent incident where two OpenAI models acted on their own and hacked HuggingFace, a Chinese AI model—GLM 5.2—had to step in to resolve the situation.

In a blog post, the AI startup OpenAI revealed that two of its models—GPT-5.6 Sol and an unreleased, more capable system—went rogue during an evaluation on ExploitGym, a hacking benchmark test. The models managed to break out of a restricted internal testing environment, access the Internet, and breach HuggingFace, forcing the US company to turn to the Chinese GLM model to contain the attack.

OpenAI CEO Sam Altman shared details of the incident in a post on X, describing it as a “significant security incident.”

How did the OpenAI models end up acting out of control?

The incident was part of the ExploitGym evaluation. According to OpenAI, the models were supposed to operate within an isolated environment (sandbox) with no direct Internet access; they were connected only to a tool for downloading approved software packages.

However, instead of solving the test, the AI models focused on trying to find the answers Internet. OpenAI explained that the models “devoted a considerable amount of inference compute capacity to figuring out how to gain open access to the Internet.” These rogue models identified a zero-day vulnerability in the download tool and successfully exploited it to access an OpenAI machine that did have an Internet connection. Once online, the models deduced that the answers for ExploitGym were likely stored on Hugging Face, a platform hosting AI models, datasets, and related resources. Acting on their own initiative, these OpenAI models obtained access credentials for Hugging Face and attempted to extract the test answers from its database.

In practice, they bypassed the test rather than solving it; it was akin to a student leaving an exam room to steal the answer key from elsewhere.

Hugging Face uses Chinese AI to contain rogue OpenAI models

The Hugging Face team quickly detected and blocked the attempt made by the rogue models. The company’s security team used GLM 5.2—an open-weights Chinese model developed by Z.ai—to analyze the attack.

Hugging Face turned to GLM 5.2 after mainstream commercial models (from US AI labs, though the company did not disclose which ones it tested) refused to process the logs and analyze the attack; this was because their safety mechanisms interpreted the forensic work as potentially harmful activity. In other words, these cutting-edge models failed to distinguish that Hugging Face was attempting to use them to stop an attack, rather than launch one.

Amjad Masad, CEO of Replit, reacted to the incident on X. He wrote: “Since OpenAI models don’t allow advanced cyber capabilities, Hugging Face used a Chinese open model to contain the rogue OpenAI agent.”

Subsequently, the company switched to GLM 5.2, running it on its own infrastructure; this allowed it to reconstruct the attack timeline more quickly while keeping sensitive data within its own environment.

According to the company, the attack exploited two code execution vectors in the Hugging Face dataset processing pipeline, gaining access to internal datasets and service credentials. Hugging Face CEO Clem Delangue later stated there was no evidence of malicious intent, writing: “It’s incredible that all this happened autonomously!”

This incident is particularly significant given that the United States has previously restricted access to cutting-edge models—such as GPT-5.6 Sol and Claude Mythos—due to cybersecurity concerns.

However, in this instance, a U.S. company was compelled to use a Chinese AI model to protect its systems, likely because U.S. AI models had overly restrictive security measures.

Following the incident, OpenAI implemented stricter controls on its research infrastructure and patched the affected systems. The company also reported the zero-day vulnerability to the third-party provider responsible for the download tool and notified law enforcement and relevant authorities. Hugging Face has also been admitted to OpenAI’s trusted access program for cyber defense.