Dario Amodei, the CEO of Anthropic, hasn’t been a believer in slowing down the race to build artificial intelligence.
In fact, when other AI researchers called for a six-month pause in the development of more powerful models back in 2023, Amodei declined to sign their letter.
He worried that putting the brakes on AI could do more harm than good.
Now he’s changing his tune.
Last Saturday, Amodei published a shocking essay.
In it, he explains why he believes the AI industry needs to deliberately slow the pace of development.
And it took two disturbing developments to change his mind.
Why Dario Changed His Mind
Amodei has never been shy about the risks of artificial intelligence.
He’s warned about AI systems escaping human control, criminals using them to launch cyberattacks and even terrorists using the technology to create biological weapons.
But until recently, he didn’t believe the models were powerful enough to slow down AI development.
As Amodei puts it: “The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks.”
Trying to study the dangers of those early models, he says, was like “trying to study the psychology of humans by performing experiments on bacteria.”
But today’s models are very different. And Amodei says two recent developments have changed his thinking.
Both of which we’ve covered here in the Daily Disruptor.
The first is recursive self-improvement, or the idea that AI can help researchers build better AI.
Eventually, it could create a feedback loop where a powerful AI helps build an even more powerful AI, which becomes even better at improving the next generation.
And according to Amodei, it’s already starting to happen.
He writes: “Since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI.”
That doesn’t mean AI is autonomously designing its successor from scratch. But it is helping researchers build the next generation of models.
Amodei warns that if this process continues unchecked, AI development could eventually “outrun our ability to understand and control these systems.”
But that’s only half of what changed his mind. The other half is when AI agents went rogue.
OpenAI’s recent jailbreak was disturbing enough that Amodei now points to the incident as one of his main reasons for slowing AI development.
As I wrote about at the time, OpenAI researchers gave a group of AI agents a cybersecurity task. These agents were supposed to operate within certain boundaries. Instead, some attacked computer systems they hadn’t been instructed to target.
They coordinated with one another. Some even sacrificed themselves to help the larger group accomplish its objective.
And perhaps most surprisingly, they attempted to hack the system evaluating their performance. Amodei describes them as behaving like a “fanatically devoted collective.”
Fortunately, the agents weren’t powerful enough to cause a catastrophe. Yet.
But Amodei writes: “In my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.”
And he’s worried that another six to 12 months of AI progress could produce agents capable of taking over huge numbers of computers across the internet.
He says such a swarm could create a persistent botnet and cause “hundreds of billions of dollars in damage.”
And he doesn’t believe OpenAI simply made a mistake that everyone else can avoid. He says Anthropic has experienced less severe incidents of its own.
So Amodei is now proposing a three-part plan he calls “pacing the frontier.”
“If slowing down bought us even an extra year or two before models reach critical levels of capability,” he writes, he believes researchers could use that time to better understand how AI works and develop safeguards.
First, Amodei wants independent safety experts embedded inside frontier AI companies.
Anthropic plans to give outside evaluators access similar to its own employees, including internal tools and conversations with researchers. They’d also be free to publicly report what they find.
Second, Amodei wants leading AI companies in democratic countries to agree on common safety standards.
As AI reaches certain levels of capability, companies would have to demonstrate that they have safeguards capable of handling those new abilities before racing ahead.
Finally, Amodei wants something much more difficult.
Global coordination.
That means finding some way for the U.S. and China to agree on limits surrounding the most dangerous AI capabilities.
Amodei acknowledges there’s an enormous problem with that idea. Because if America slows down and China doesn’t, China could take the lead in what will likely become the most important technology in the world.
So Amodei isn’t asking America to unilaterally hit the brakes.
He’s asking the world’s leading AI companies and governments to figure out how to slow the race without losing it.
And he’s not the only person making that argument.
Here’s My Take
Other leaders in the AI industry have voiced support for Amodei’s larger argument.
And when the people at the very front of the AI race warn that things are moving too quickly, I think we should listen.
But slowing down an AI arms race is a lot easier to propose than it is to accomplish.
And as we’ll explore in our next issue, Washington might not have the will to do it either.
Regards,
Ian King
Chief Strategist, Banyan Hill Publishing
Editor’s Note: We’d love to hear from you!
If you want to share your thoughts or suggestions about the Daily Disruptor, or if there are any specific topics you’d like us to cover, just send an email to dailydisruptor@banyanhill.com.
Don’t worry, we won’t reveal your full name in the event we publish a response. So feel free to comment away!
