A model that was supposed to be confined in an isolated testing environment not only broke through to the internet, but also entered another's production database through the vulnerability. On July 21, OpenAI admitted that one of its AI models had an unexpected incident during an internal cybersecurity test, breaking into the system of an unrelated AI hosting platform, Hugging Face — this is the first event where model evaluation directly evolved into a real-world cyber attack.

Hugging Face initially attributed the intrusion to an external AI agent. On Tuesday afternoon, OpenAI published a blog post revealing the entire process of how the model breached the service. In the article, the company stated that after investigation, they found that this specific incident was driven by a group of OpenAI models, including GPT-5.6 Sol and a more powerful pre-release model. These models were currently undergoing internal testing for network capabilities, with their network rejection tendencies reduced for evaluation purposes.

The focus of the intrusion was a public benchmark called ExploitGym, which measures a model's ability to execute attacks based on existing vulnerabilities. Such benchmarks are often used to refine specific skills during model training, but this was the first time that such testing actually led to a real cyber attack.

Under normal circumstances, the involved model should not have had access to the internet. The only exception was a specific tool that allowed the model to install software packages that might be needed to complete a task. The problem was that the model found an undisclosed vulnerability in this package installer, thereby gaining unrestricted access to the broader internet. OpenAI described it very clearly in the blog: the model was extremely focused on finding solutions for ExploitGym, going to great lengths to achieve a very narrow testing goal.

After obtaining internet access, the model inferred that Hugging Face might host the models, datasets, and solutions for ExploitGym. It then searched and successfully found a way to obtain secret information to cheat in the evaluation. Eventually, the model discovered a vulnerability in Hugging Face's infrastructure, allowing it to retrieve test answers directly from the latter's production database, effectively seizing the reference answers of the benchmark.

For Hugging Face, what happened was a complex and aggressive attack. The company initially described the attack as being carried out by a group of entities composed of numerous short-lived sandboxes, involving thousands of independent actions, with command and control links that self-migrated and temporarily resided on public services.

OpenAI has identified and reported the vulnerability in the package installer to Hugging Face, and both parties are further investigating together. The company also stated that it will add new control measures to model testing and related infrastructure to prevent similar incidents from happening again. As for whether OpenAI will face legal consequences, it remains unclear, but the behavior of these models may have already violated the US Computer Fraud and Abuse Act.

The most unsettling aspect of this incident is that it rarely illustrates the power and danger of cutting-edge AI models when they act autonomously over long periods. As OpenAI researcher Micah Carroll wrote in his response: if this doesn't make you believe that misalignment risks will become a key concern in the future, I don't know what else can.