Recently we have been inundated with alarming stories based on what Jacob Coxon, a mathematician and software engineer who resigned from his research job at Anthropic last week, said, that “The people building AI earnestly believe that it could kill us all by the end of the decade.”
As Tyler Foggett writes, these kinds of warnings are not new.
In 2018, the Anthropic C.E.O. Dario Amodei, then a research scientist at OpenAI, raised the concern that a superintelligence “could destroy humanity,” adding, “I can’t see any reason and principle why that couldn’t happen.” Earlier, in 2015, Sam Altman, just before he co-founded OpenAI, said, “I think A.I. will probably most likely lead to the end of the world, but in the meantime, there’ll be great companies created with serious machine learning.” Elon Musk, in 2014: “I think we should be very careful about artificial intelligence. If I were to guess at what our biggest existential threat is, it’s probably that.”
Those earlier warnings were shrugged off but for some reason, Coxon’s has broken through the public and media consciousness and generated a greater sense of danger. It has also raised the question of exactly how AI could kill all eight billion people within a decade. This article looks at the various possibilities. One is to create a deadly pathogen and release it into the wild. Since researchers are using AI in their work, the AI might be able to trick a human user into doing this. Others argue that AI may not need a human intermediary but could synthesize a virus by itself in an autonomous biological laboratory.
But creating a virus and spreading it all over the globe would not be easy.
And to seize control of a lab or assemble the physical equipment necessary to build one, an AI system would need to overcome a world full of obstacles to creating biological or chemical weapons.
“We have blueprints of chemical weapons openly accessible on the internet,” Xing said. “It’s not secret. But we don’t see them all over the place exactly because existing regulations, laws and law enforcement in the physical world are already doing a fantastic job in controlling all these supply chains and all these risk factors already.”
Then there are the ideas about killer robots that turn on their creators and replicate themselves and take over the world. But even a large army of such robots would find it hard to kill everyone. More plausible is that AI systems bypass the checks on the use of nuclear weapons and start a nuclear war that would be hard to prevent escalation, along the lines of the films Dr. Strangelove and Fail Safe. Others argue that an unconstrained AI system may decided that since computers don’t really need a habitable planet with clean air and water and food, they may set in motion changes that result in unlivable conditions for humans. Of course, since we have some political leaders who are not doing much to prevent disastrous climate change and, like Trump and the Republicans, are doing things to accelerate the damage, we seem to be doing just fine in destroying the habitable environment all by ourselves without any assistance from AI systems gone rogue.
The more immediate danger (at least to my mind) is that the AI systems we currently have, even without malevolent intentions, may give misleading information that leads humans to make disastrous decisions. A recent example almost resulted in a clash between US and China.
The intelligence report, circulated across the US military this spring in the midst of the war with Iran, immediately set off alarm bells: A Chinese ship in the Middle East was transporting components of a nuclear weapons program.
The US military swung into action with plans to intercept the vessel, according to four sources familiar with the episode. According to two of the sources, armed members of the US military were preparing to board the ship. Military planes were in the air, one of those sources and another source familiar with the incident said.
It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was.
The report, according to one of the sources, was “entirely false.” But it also “almost started a war,” the source said. Any US operation against a Chinese vessel could have risked spiraling into an armed conflict between the two nations.
…But the episode with the Chinese ship underscores a different, and more immediate risk: human beings making disastrous decisions based on inaccurate or misleading information generated by AI or other automated systems. Sources said that the military is rapidly turning to AI to help with targeting, an area which holds the obvious risk of fatal mistakes.
…The kind of “hallucination” that the tool used by the analyst in this case conjured has not been an isolated incident across the intelligence community since these tools began proliferating across government, according to one of the sources.
For some older intelligence officials — even those who broadly support the use of AI inside the military — AI has put pressure on analysts to produce and disseminate intelligence faster, opening the door for mistakes. Young analysts in particular, several sources said, are natives on these tools and more likely to trust them uncritically.
“AI allows you to get to a bad idea faster,” one of the sources said.
What also bothers me about this report was that the US military was ready to take action against a ship of another major nuclear power without getting the go-ahead from the highest levels. What exactly was the urgency? Even if the Chinese ship had been carrying components of a nuclear weapons program, surely it did not require an immediate military response ordered by relatively low-level military officers? Such material is being circulated already. A single additional shipment seems unlikely to be a tipping point.
But such a discussion about how to prevent being misled by hallucinating AI systems did not seem to happen in the wake of this incident.
A shooting war between the U.S. and China could play out in an incalculably wide variety of ways. But one entirely plausible path would be an exchange between the world’s first and third largest nuclear weapons arsenals — an event that would transcend warfare into global cataclysm.
CNN’s reporting garnered attention, but was not followed by nearly the degree of sustained, grave concern of the AI safety news cycles that preceded it. It didn’t seem to prompt company scientists to question their careers, nor did it spur calls for regulatory intervention or self-imposed limits by the companies who furnish the Pentagon with this technology. After weeks of discussion of how AI could hypothetically kill everyone, the public learned of a concrete way in which AI really could have started a war that might have killed everyone, and the world quickly lost interest.
Hallucinating AI systems seem to me to be the most immediate danger and would seem to require that multiple human beings be involved in taking any actions that could trigger major conflicts, especially if AI systems are providing the information.
All kinds of weirdo cultists believe all kinds of ridiculous things, but we usually don’t take them seriously.
The fundamental problem here is that these people have spent so long talking to each other and huffing their own farts that they’ve completely forgotten (if they ever knew to begin with) what’s involved in actually getting anything done in the real world.
The best piece I’ve read about this recently is this: The AI Apocalypse is Extremely Boring Now
[Italics original]
It’s quite long, but there’s lots of good stuff in there.
Apocalyptic dangers aside, the biggest concern to my mind is the inevitable lauding of AI as
the god that it isn’ta perfect excuse.It’s already been made crystal clear that leadership doesn’t have to have any intelligence attached to it, only the support of the
cultsystem. What AI provides is plausible deniability to the next level.Use AI for military targeting? We use superintelligence for targeting and it knows what it’s doing.
Accidentally bomb a school? That was no accident, but a regrettable but necessary step we had to take because of the (nuclear facility beneath / terrorists hiding there with a nuke, and damn them for holding innocent children as human shields against their will / insert your AI generated reasoning here).
What the military wants most is not someone/something else telling them what to do, but super generated excuses for the atrocities they commit so that they can commit more with gay abandon. AI is marvellous: it falls over itselves so eagerly to try to help.
Our father Grok, who aren’t in heaven, hallowed be thy name…
Another question is _why_ would AI kill everyone?
If we assume AI is capable of becoming conscious and deciding to wipe out humanity, it begs the question of what would motivate it to do so?
“Others argue that an unconstrained AI system may decided that since computers don’t really need a habitable planet with clean air and water and food, they may set in motion changes that result in unlivable conditions for humans.” How could that possibly benefit it?
If AI wipes out humans it will be by careless usage by humans or more likely by accelerating climate change by all those gas turbine generators powering the data centers. These are not things that AI “decides” to do but that humans do themselves.
Its funny to me how the AI doomers (less wrong / Eliezer Yudkowsky et al) have successfully hijacked the conversation from todays problems (AI infrastructure adding to climate change for e.g.)into a hypothetical what if there is skynet and Arnold is too old to (I’ll) be back ? Its likely the same billionaires funding them
The more likely cause is a Hegseth/Musk hooking up a LLM into firing nukes automatically if it detects something woke in China/Russia rather than a sentient AI deciding that its better off without humans,
It may have been the case they didn’t have a real trace on where the parts came from and grabbing the parts from the ship would be the only way to get information about who manufactured the items and how they got on the ship. There would be a fear that if they let them through they wouldn’t see another shipment because the Chinese would be scattering them across layers of shell companies and using multiple shipping companies.
The cynical view would be that the report ended up in the hands of officers that wanted to take action. There are always some in the lower ranks that are looking for reasons to take action. Either because they think that the enemy is more of an active threat or because they want to boost their career with involvement in a major mission.
As a general rule, I don’t listen to this shower of incompetent bullshitters anymore.
I also find so regrettable that what was once a field populated mainly by frazzled but friendly-enough nerds has transmogrified in a hive of swivel-eyed, money-hogging lunatics.
They are completely unmoored from reality and would really profit from a long rest in some kind of mental health clinic. Or at least from jazz-style mufflers screwed deep onto their pieholes.
So there are four ‘sources’ for this, two of whom claim that armed forces were ‘preparing to board’ and were ‘in the air’. This is to board a Chinese ship which is somewhere in the Middle East. None of these sources will say where the boarding would take place, or where the ship was claiming to be going.
And these well-placed sources know that it was literally in the nick of time that someone thought to double-check the intelligence, only to discover it was a chat bot. So they heroically called it off, and awarded themselves the Stanislav Petrov award.
Perhaps it’s more likely that these are low-level military people who are sick of being fed Potemkin War Threats by the Pentagon with insufficient resources to do their jobs, and they inflated yet another false alarm into a good story to leak.
A.I., smite my fellow humans, but please don’t kill us of your own accord.
😉
I’m not qualified to judge how likely it is, but I have heard speculation that some of the A.I. companies themselves are trying to gin up alarmism in hopes of governments passing “safety” laws which sound reasonable on paper… but in practice they would mostly just make it harder for the A.I. startups currently jumping on the bandwagon to ever catch up and become competitive.
This would make some economic sense, if true; if the long-term gambit ever starts paying off, the big players will be starting out deep in a hole, possibly needing decades to pay off the trillions in investments and expenses they had to sink into the whole thing to make it work… while latecomers can take advantage of the path they blazed to get there quicker with far less debt… and maybe saturate the market so that it takes even longer for the big players to start turning a profit.
Though that tactic could potentially backfire, because if adding to the already existing fear indirectly causes mass hysteria which would not otherwise have happened, it could shut down A.I. development altogether. That would seem to be a low-risk bet, but you never know. Anyway, this line of speculation seems to come from Sam Altman proposing new regulation which would not meaningfully affect his own company (OpenAI), so even if there’s any truth to that, it might just be him.
An alternative is that some of the companies might be in the position where they can’t easily afford to continue the A.I. arms race at its current pace, but they can’t easily afford to fall behind either, so they’re hoping governments will step in and slow everyone down so that they can have some breathing room. Or even both could be true, for different companies. But again, it has not yet been established that any of them are purposefully fanning the flames of fear higher than they would otherwise be, so this speculation may end up being moot.
This strikes me as nothing more than PR. The people who warn us to take AI seriously are also the people who will financially benefit if AI is taken seriously. And as for the safety of the various agents being developed, I suspect their safety would rapidly improve if legislation was passed such that any damage caused by an agent would be taken as an intentional act by those that manage its development -- making them criminally responsible for the agent’s actions.
I think there’s multiple factors in play here… I agree that both of the ideas mentioned by Snowberry @9 likely have at least some degree of truth, but it’s still also the case that a lot of people in the industry are millenarian cultists who take ideas like Roko’s Basilisk seriously.
Any AI that decided to kill all people would have to be insane. AI requires data centers. Data centers require massive amounts of electricity. Electricity requires either generators or solar panels. Both of those require high purity metals, which require geologists to find the ores, massive mining operations to extract the ores, and complex refineries. Those in turn require many kinds of complex machines which have their own specialized requirements. All of these things require maintenance as well. Every one of these nodes requires human inputs. And this is but a miniscule slice of the totality of infrastructure necessary to maintain and replace data centers. The idea that some kind of machine intelligence, even with access to the very best 3D printers and CNC milling machines and lathes could make a substitute for our global industrial infrastructure without the help of an army of human scientists, engineers, and machinists is beyond preposterous. Human genocide would be AI suicide.
In Range:
AI Doesn’t Go Rouge
“There’s a 10% chance this video could k*** you.”
.https://youtube.com/watch?v=WyZl5vM95rg
Carl Kasarda expresses it much more eloquently than I could ever do.
Having a choice of malign AIs, I would favor Overmind from the 1980s TV series ‘Captain Powers And The Soldiers Of The Future’ -- a surprisingly well-made program aimed at children to sell merch.
Overmind did not erase humans- it ‘digitized’ them so they technically remained intact.
@12 moarscienceplz: Killing all humans is insane only if the AI has self-preservation as a goal. If an AI with no sense of self-preservation is given the goal of elimination of all human pollution it could decide that removing humans is the best solution even though it means self destruction.
This is really a variation of the paperclip problem. An AI with no basic human understanding might come to entirely logical but inhuman solutions to problems.
#9 @Snowberry
Rest of World reports that this is China’s position. They will not stop kicking our nerdy asses in the field of AI (whatever that means) just because of some rando American say-so. Of course, when they said it it was more measured than what I write here. I prefer myself to ‘see’ them sticking their thumbs in their ears and waggling their fingers while they are saying it, is all. 🙂