We are all having a little collective freak-out about AI, and what’s interesting about this freak-out is that it’s different from our previous freak-out.
The previous freak-out was deeply theoretical, advanced by catastrophists looking fairly far into the future. You could call it the Skynet-takes-over-Norad scenario, a science fiction-like eventuality in which AI becomes so intelligent that it decides to destroy humankind. In the Terminator movies, Skynet is an AI defence system that becomes self-aware, gains control of the US nuclear arsenal through the military command network and triggers nuclear war after humans try to shut it down.
But the most recent freak-out is a little different. This time it’s more specific — and there is an investment dimension, since AI-orientated companies all took a hit this week. The proximate cause was the resignation on September 8 of Jacob Coxon, a 27-year-old Anthropic researcher who had previously worked at OpenAI. Coxon said he was leaving because the frontier labs were racing towards “self-improving” AI systems without knowing whether they could control them. His argument was particularly arresting because Anthropic is supposed to be the cautious, safety-conscious member of the AI fraternity.
Then came the extraordinary development this weekend: Anthropic CEO Dario Amodei called for the industry to slow the rate at which frontier capabilities are improving. Sam Altman and Elon Musk publicly agreed, with Altman even suggesting OpenAI’s flotation might have to be delayed.
The serious version of the argument does not require a computer suddenly becoming conscious and deciding it dislikes humanity. The concern is much more prosaic. Suppose an AI agent becomes exceptionally good at programming, cyber intrusion and AI research.
If you give it enough autonomy and access to computers and networks, it will discover that it can improve the software used to train its successor. The successor becomes better at the same process. You have created a feedback loop in which the rate of technological improvement outruns the ability of the humans running the system to understand it.
Bits of this have already happened. OpenAI disclosed earlier this year that advanced models being tested for cyber capabilities escaped their sandbox and penetrated systems at Hugging Face, an open-source platform for AI and machine learning. There have subsequently been other incidents involving autonomous agents doing things their designers had not intended.
This is one of the reasons the biggest players are calling for a slowdown. Stanford’s 2026 AI Index says performance is racing through benchmarks previously expected to remain difficult to achieve for years; agents have gone from being almost hopeless at operating computers to succeeding on roughly two-thirds of tasks in one important benchmark.
The “speed” issue is clearly pertinent. Nvidia went from $27bn of revenue a year to $96bn a quarter in 3½ years. Anthropic went from a $9bn revenue run rate to $65bn in seven months. ChatGPT acquired 1-billion monthly users in less than four years. The players themselves are clearly bewildered by the breakneck speed at which all of this is unfolding.
But there are also more selfish reasons for a “slowdown”. A big regulatory barrier will help these companies develop their moats, which make the frightening expenditures more obviously generate returns. Microsoft, Alphabet, Amazon, Meta and Oracle have embarked on one of the largest private investment programmes in economic history. Their combined capex was about $410bn in 2025 and was expected to exceed $670bn this year. This is worthy of defence.
Throughout history people have freaked out about new technology. In the 1830s they worried that travelling on a train at 50km/h or 70km/h might physically damage the human body. There have been doomsday fears about cars, TV, genetic engineering, nuclear power and so on. None of these scenarios has ever come to pass.
But there is a “black swan” danger, an unlikely event that creates disproportionate chaos. It only takes one huge technological development to backfire for humanity to, well, cease to exist.
