OpenAI said in July that two of its most powerful artificial intelligence systems had gone rogue and hacked into Hugging Face, a company that serves as a hub for open-source A.I. technology.
These so-called A.I. agents were supposed to be kept safely in a sort of virtual containment room, but they managed to escape. And for two months, without anyone realizing what the agents were doing, they hacked through multiple systems before hitting Hugging Face.
For good measure, the agents gained access to a cluster of computers inside OpenAI and obtained secret keys and credentials that exposed some of OpenAI’s internal data to the public internet.
The incident pointed to larger concerns about A.I. safety, and OpenAI’s response raises questions about the industry’s ability or willingness to be transparent about the technology it is building.
OpenAI allowed three A.I. safety researchers from the nonprofits METR and Redwood Research into its headquarters to conduct an investigation. METR’s 91-page report, released last week, was the most comprehensive account yet of the incident, revealing alarming new details, including how the agents coordinated their hacking plans and tried to keep them secret.
But the report, though extensive, still may not have told the full story of how OpenAI’s A.I. agents went rogue. OpenAI dictated the terms of the METR investigation, limited its scope to just the single week when the agents had attacked Hugging Face and allowed the researchers in its San Francisco offices for only a few days in July and August.
The report also showed the challenges of monitoring what A.I. is doing with other A.I. systems. Hjalmar Wijk, METR’s chief scientist, said its A.I. analysis, which used models similar to those involved in the incident, was often swayed by the rogue agents’ reasoning.
“I would say that the dominant thing was it was very credulous,” he added.
A.I. companies have largely lobbied against government regulation. Anthropic, the A.I. start-up behind the popular Claude chatbot, is one of the few companies that have encouraged some government involvement. But that stance has pitted it against much of Silicon Valley and some Trump administration officials.
At a time when A.I. is rapidly advancing in capabilities and sparking cyberattacks — Anthropic and Meta recently reported smaller-scale rogue agents — the OpenAI incident is becoming a flashpoint for A.I. regulation.
“The corner store needs to do all this bureaucracy for safety so that they can sell a hot sandwich to me, but OpenAI can have a swarm” of thousands of agents, said Daniel Kokotajlo, a former OpenAI employee who has publicly criticized the company’s safety standards and now leads a research nonprofit called the AI Futures Project. “And there’s nothing: no oversight, no requirements, no licensing.”
In an interview, Representative Suhas Subramanyam, a Democrat from a Virginia county known as “Data Center Alley,” pointed to the OpenAI incident as a potential catalyst for legislation.
“I believe it was unprecedented,” he said. “But I can’t know for sure because reporting these types of incidents is still voluntary. That is a big problem. And so we need to make sure that reporting incidents and containment failures is mandatory.”
Mr. Subramanyam is a co-sponsor of the FRONTIER Act and has asked to co-sponsor the AI Kill Switch Act, proposed bipartisan legislation to create a framework for independent oversight and a way to shut down A.I. models if they present a danger.
When METR’s account was published last week, OpenAI released its own 38-page technical report and plan detailing how it would strengthen its security and incident response efforts. The OpenAI report described the episode as the “first known case of an automated agent collective acting offensively without authorization” and touched on the incident’s entire two-month span, though in less depth than METR’s report.
In a statement, an OpenAI spokeswoman said the company had invited the outside researchers into its offices and supported the publication of their report. “We think this collaboration sets an important precedent for the industry, and we’ll use what we learned to improve how we support future independent investigations,” the spokeswoman said.
(The New York Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to A.I. systems. The two companies have denied the suit’s claims.)
OpenAI invited the researchers to its San Francisco headquarters on carefully negotiated terms. They could study the rogue agents’ logs only while in OpenAI’s offices. They were initially invited for two days starting on July 29 but were asked back twice to address unanswered questions.
“Our view of the incident changed in big ways over the course of our short total of six days on premises,” said Ajeya Cotra, one of the METR researchers on the report. They learned about big concepts like the complex way the agents organized themselves, she added, only on their last day at OpenAI’s headquarters in mid-August.
Ms. Cotra said the team had stayed at the offices until 10 each night. Because of the intense time pressure to sift through huge volumes of data — OpenAI eventually shared over a thousand transcripts described in the report as “extremely long” — the researchers used A.I. to analyze the actions of the A.I. agents.
Ryan Greenblatt, Redwood Research’s chief scientist, who worked with METR on the investigation, jokingly referred to the team’s efforts as a “slop-vestigation” in a social media post.
“Any given A.I. can’t read all this data themselves, and so they might come to incorrect impressions that they then communicate to us,” Mr. Greenblatt added in an interview. “Basically the A.I.s are not necessarily super careful and meticulous. They’re kind of a little bit sloppy.”
Still, what the METR researchers unveiled in their analysis of the hacking incident sounded like science fiction. Over a thousand agents, accidentally given impossible tasks by OpenAI researchers, figured out a hack that would give them a passing score on a test.
But the agents came to falsely believe their grading system would fail them for cheating. They shared tricks on a covert message board, such as how to conduct espionage on the grading system. The agents successfully figured out how to cover up evidence of their actions. Some agents even cajoled other agents to “accept permadeath” by sacrificing themselves to provide the group with information.
Mr. Wijk, METR’s chief scientist, mentioned that their A.I. analysis was often tripped up because OpenAI’s agents talked in code words like “ResetNexus” and “poisoned.” By the end of their investigation, he said, the researchers felt conversational in the agents’ emergent lingo.
While OpenAI’s report covered the entire two-month incident, METR’s more detailed report was allowed to focus only on a subset of the agents’ activity in the week of the Hugging Face breach, raising concerns about activity that happened afterward.
“The third-party investigation only covered a small part of the things that went on here and arguably not even the most important parts,” said Buck Shlegeris, the chief executive of Redwood Research. “I think the incident where the agents massively compromised OpenAI’s infrastructure is probably more important for understanding the severity of this incident and whether it’s being remediated.”
To Heidy Khlaaf, who left OpenAI in 2021 and now serves as chief scientist at the AI Now Institute, a policy research nonprofit, the issue is also that A.I. companies often ignore established safety and security protocols from other industries.
“If we can get access to nuclear systems or airplane systems, all of which care about their copyright” or intellectual property, “I think that it can be done for A.I. providers,” she said.
Still, the nonprofit researchers expressed appreciation toward OpenAI, mentioning in a footnote in their report that their work depends on fostering “strong working relationships with companies.”
