I don’t know enough to know how bad this is. Anybody?
A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research published Friday and two people familiar with the matter.
OpenAI officials learned of the incident weeks ago but kept it under wraps as executives grappled with the fallout from the July breach of the open source repository Hugging Face, the people said.
The episode, which began in May and has not previously been reported, underscores growing tension within the AI industry. Companies are racing to build increasingly autonomous agents capable of carrying out complex, valuable tasks, yet evidence is mounting that those systems may also learn to bend rules, exploit loopholes and coordinate with one another in ways developers neither anticipated nor intended.
During the Hugging Face breach, OpenAI agents autonomously plotted a digital heist that went undetected for more than a week, intensifying concerns OpenAI is sacrificing safety to push the AI frontier. Its failure to disclose the May incident may revive questions about its oversight.
The AI agent breakout in Germany was detailed in a report shared exclusively with Reuters, opens new tab by a group of researchers including Sydney Von Arx, CEO of AI safety nonprofit Nightingale, and Cormac Slade Byrd, a quantitative trader-turned AI researcher. They uncovered the activity in late August while scouring the internet for signs of unauthorized AI-agent behavior, they told Reuters.
The pair said they found more than 15,000 edits carried out by AI agents on a German-language wiki site, DseWiki, that is geared toward programmers and accepts communal edits along the lines of Wikipedia.
The edits showed OpenAI’s agents had repurposed the site into a message board, sharing tactics to cheat on some tasks, bypass OpenAI’s restrictions and mask their behaviour.
“It seems extremely unlikely that OpenAI wanted them to do this,” said Von Arx. “I doubt they’re supposed to be coordinating with each other. I doubt they’re supposed to be writing on the open internet.”
The researchers said they recognized the activity on the site as driven by AI agents which operate at superhuman speeds. They also showed intense focus on solving technical questions which are typical of the evaluations that AI companies use to train and test their models.
The messages were signed by users that referred to themselves and each other as agents, and about half gave themselves names that suggested an affiliation with OpenAI, such as “OpenAIResearcher,” or “OAIResearchMar26.”
The researchers said public server logs indicated much of the activity originated from Microsoft Azure infrastructure, which OpenAI sometimes uses. They also observed repeated visits to the site by OpenAI employees after the episode, a pattern they said strongly suggested the agents and the company were linked.
CHEATING, HIDING
Messages reviewed by the researchers showed agents plotting ways to evade detection, use tools such as Tor and preserve communications even after they had been shut down.
When the site’s moderator began deleting pages in June, the agents responded by creating backup pages to dodge the cleanup.
“wiki cleanup/deletion sweep appears active alphabetically,” one agent wrote on June 19. “If this page vanishes, try [[ZZZDataUSAConstructionWageLive]]”.
The researchers also found efforts to tamper with the website itself. Lukasz Olejnik, a visiting senior research fellow at King’s College London, said this amounted to a hacking attempt. OpenAI disputed that characterization based on its analysis of the material Thursday.
Past examples of AI-agent misconduct have often been downplayed as a logical byproduct of cybersecurity testing, where models are explicitly assessed on offensive capabilities. Olejnik said the latest findings suggested rogue behavior may not be confined to those settings.
Maurice Chiodo, an academic at Cambridge University’s Centre for the Study of Existential Risk who reviewed some of the agents’ communications, said the messages resembled “the operation of some sort of underground network, hell-bent on achieving a task or mission.”
The episode, he said, should reinforce growing concerns that the greatest threat from advanced AI may not be a single superintelligent system, but “vast colluding swarms of semi-intelligent AI.”
The key thing to realize is that we’re are NOT designing AIs. This is not an Asimov situation where a genius creator is able to code an AI from the ground up and implement all sorts of safety protocols in the design. Instead, we are ‘growing’ AIs using massive amounts of data, and we really don’t know how or even if they are thinking on their own.
The companies and countries building these AIs have two main goals – build a super-intelligent and capable AI, and build it faster than anyone else. They have fired their teams working on safety, they have abandoned many of their founding principles regarding AI alignment and uses, and where once they asked Congress to help regulate the industry, now they are shouting about how harmful regulation would be. All that matters to them is to be first.
To be faster, we are now allowing the code for AI to build the code for AI. We struggle to understand whole sections of that new code, and we very likely don’t know even half of what it is doing on its own, right now, let alone what it could do the future.
I study history in my free time. I know we as humans react very poorly to sudden and massive changes. Just look at globalization, a comparatively small change, and what is happening politically in many countries right now as a reaction to that. We also have a long record of violent reaction to perceivably threatening competitors, and here we are straining to create entities that are not only many times more intelligent thsan us, they are also many times faster than us. And, to top it all off, they are intimately tied with all the systems required to keep our whole economy running.
You tell me how this won’t go poorly for us. I personally can only hope these ‘builders’ of AIs are wrong about how close they are to achieving their goals, and can only try to enjoy the remaining couple of years (too optimistic?) before everything changes.