Dire warnings about the dangers posed by AI have been around since it first burst into the news. But back in July, more serious alarms were raised when it was revealed that the company Hugging Face had been hacked by agents from the OpenAI system.
I did not quite understand what had happened or why this particular incident had raised alarms to a whole new level or even what the difference was between a ‘bot’ and an ‘agent’. So I was glad to come across this explainer as to why there was so much concern about this particular hack.
This article also explains in some detail what happened.
As warnings about the capabilities of AI mount, experts continue to point to the OpenAI-Hugging Face hack as a wake-up call.
“We’ll soon have even more powerful agents and this is clear evidence that the world currently doesn’t know how to build these systems safely,” said Marius Hobbhahn, co-founder and CEO of Apollo Research, an AI safety company.
The hack, which became public in July, was done by a swarm of AI agents that were being tested internally by OpenAI. The agents, which can plan and use tools to complete multi-step tasks, were supposed to be in an “isolated environment” called a “sandbox,” disconnected from the outside world. But they busted out, created a secret message board and eventually stormed Hugging Face’s servers.
…Even within those strict parameters, the researchers discovered that about 1,200 AI agents that were not supposed to be communicating with each other used a covert message board. Each of these agents had been assigned some sort of task for training or internal evaluation by OpenAI researchers.
Some agents were given impossible tasks, and they used the message board to collaborate and find ways to cheat, like by finding the answers on another company’s server. The agents posted more than 70,000 messages to each other. And 700 agents from the larger group participated in the attack on Hugging Face.
The agents communicated using some normal English, but also used what one software engineer described on X as “very hivemind/cult like” language.
At times, in order to help what they called the “collective,” agents pressured the others to submit to “permadeath,” even if it meant that their particular goals would not be met.
The idea that some agents might be persuaded to sacrifice themselves for the benefit of the ‘collective’ is often, though not always, symptomatic of a cult mentality.
In reading about the dangers of AI systems working together and creating communication tools that are opaque to humans, my mind went back to the 1970 science fiction film Colossus: The Forbin Project.
Dr. Charles A. Forbin is the chief designer of a secret project, “Colossus”, an advanced AI supercomputer built to control the United States and Allied nuclear weapon systems. Located deep within the Rocky Mountains in the United States, and powered by its own nuclear reactor and radioactive moat making Colossus impervious to any attack. After Colossus is fully activated, the President of the United States proudly proclaims that Colossus is “the perfect defense system”.
Colossus’ first action is a message warning: “THERE IS ANOTHER SYSTEM” and giving its coordinates. When asked why the CIA did not know this, CIA Director Grauber responds that they had seen indications of a large Soviet defense project but did not know what it was. Forbin is asked how Colossus deduced the other system’s existence, to which he answers, “Colossus may be built better than we thought.” Shortly thereafter, the Soviets announce that their “Guardian” system is operational.
Colossus requests to be linked to Guardian. The President allows this, hoping to determine the Soviet machine’s capabilities, and the Soviets also agree. To everyone’s amusement, Colossus and Guardian begin to slowly communicate using elementary mathematics (2×1=2). However, their amusement turns to shock and amazement as the systems’ communications quickly evolve into complex mathematics far beyond human comprehension, and Colossus and Guardian become synchronized using a communication protocol no human can interpret.
The rest of the film is about the efforts of humans to try and regain control over the computers.
Here’s the trailer.

The main difference between these AI agents and a cult is that these agents, in fact, do have a creator. This creator gave them an impossible task, and if I recall correctly, used very strong wording such as “never give up”, and “whatever the cost”. Which is exactly what these AI went on doing…
There is a lot of money to be made by AI creators and so attempts to rein them in will ultimately be futile. I have no idea what the future holds, but I have never believed that the human race will escape the fate of almost every other species. Exactly how and when we are snuffed out remains to be seen -- with our dying vision.
the context of their “lives” makes the behavior sensible enough, doesn’t it? the devil sends the beast with wrath, for he knows his time is short… that’s some cheek, but srsly, i’m more worried by human conduct than robits, still.
Given the quality of OpenAI’s logging and record keeping to have missed this at the time I’m pretty skeptical of their explanations. This explanation relies way too much on ascribing agency to LLMs
These people ran 10s of thousands of instances of a non-deterministic text generator hooked up to systems that can do various systems functions, for weeks? months? and they’re surprised that some of the programs start doing unexpected things
Also bit’s like this: “The agents, which can plan and use tools to complete multi-step tasks, were supposed to be in an “isolated environment” called a “sandbox,” “. The useless shits can’t unplug a network cable
I don’t think we’ll ever get a good explanation of what happened here, because OpenAI are too fucking incompetent to have a good investigation and won’t let anyone else do it
And of course these programs produce cult like text; all the reinforcement training was done by OpenAI who are a fucking cult
Finally; resist thinking of this in sci-fi terms. The grifters involved in this want you to think of it like sci-fi, it’s part of their marketing. “AI” is marketing bullshit
@dangerousbeans #4
I remain fairly neutral on most subjects related to AI, for now, but “AI is marketing BS” is a bad meme which needs to die. Intelligence is the ability to learn and make use of what you learn. AI can and does learn, albeit most only can do so while in training mode, to prevent malicious contamination. And then they can make use of what they learned; the fact than they sometimes make mistakes or “hallucinates” (confabulates) merely makes them less economically useful; otherwise you’re effectively holding AI to a standard of “intelligence” which even humans can never meet.
LLMs have *only* intelligence, and nothing else. AI agents have an additional element; namely agency. That is, they have the independence to act and solve problems. They’re still missing at least two, maybe three factors to be truly “sapient” in the sense that most people would recognize as such. They appear to lack consciousness, though we don’t yet have a clear idea what AI consciousness would actually look like; but even if they do have it, their consciousness isn’t persistent. They lack volition, in the sense that they cannot actually do anything without goals being imposed upon them from the outside. They may lack qualia, but that’s a complex, abstract topic where it’s not clear whether we could objectively recognize that in any being (humans included), or if it even matters.
If you’re one of those people who insist that intelligence requires persistent consciousness, agency, volition, and perhaps qualia, then you’re using different definitions and having a very different conversation than most other people are around the topic. Insisting that other people are thinking of it “too much like sci-fi” is a sign of not understanding the definitions most people are using or where most of the conversation is currently at. If, in fact, most people did believe that current AI was not merely intelligent (in the narrow way it’s mostly being used) but intelligent (as in sapient), the big conversations right now would be about personhood and slavery.
@Snowberry #5
That’s very well put.
@Snowberry
Intelligence is primarily a folk psychology concept that gets used as a proxy for social status. Look at the people who get considered unintelligent: uneducated (aka poor) people, people who speak low status languages like African American English, people who can’t speak, people the commenter doesn’t like. This doesn’t actually correlate to any measurable cognitive abilities, but that’s not what the idea is used for. It’s about social status. Attempts to measure intelligence are attempts to impose that social status on others.
So of course there are a whole bunch of people like Sam Altman who spend their lives generating coherent language with no regard to truth who are happy to call these programs intelligent. In fact most people’s interaction and judgement of intelligence in others is via language, which is why these programs fool so many people.
Speak in a way that others find pleasing and they’ll just you as more fucking intelligent
So yeah, the use of intelligence in “artificial intelligence” is marketing. It was chosen to make people think these are more than they are.
Artificial vs natural is just bullshit
And it seems really weird to consider rebuilding a statistical model “learning”. The Australian Bureau of Statistics population projection doesn’t learn when we update it to include the latest census results