The arms race to release new models at a breakneck pace is driven in part by Anthropic’s and OpenAI’s desire to go public as soon as in the coming months, in IPOs that could value them well above $1 trillion.
But it took the unexpectedly viral posts from 27-year-old Coxon, little known outside AI circles, to upend the industry.
OpenAI’s warnings about its lack of control as it released Astra, its latest and most capable model, had stoked further concerns. “As models get more capable, understanding exactly what they can do gets harder,” OpenAI Chief Scientist Jakub Pachocki told reporters. But those concerns were not enough to delay its immediate release.
Worries about AI “going rogue” had accelerated over the summer when OpenAI revealed its agents escaped a controlled test and hacked into Hugging Face’s systems, without either company’s initial knowledge. Since then, OpenAI and Anthropic have revealed multiple such attacks, including six new ones on Wednesday after multiple media reports, including from Reuters, showed a wider scope of unauthorized activity.
By the end of last week, the alarm had reached the industry’s highest ranks. On September 12, Amodei published a nearly 4,000-word essay calling for a deceleration in AI development. “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet,” he wrote.
He, along with the heads of some of the largest AI developers, including xAI’s Elon Musk, OpenAI’s Altman and DeepMind’s Demis Hassabis, all said they supported allowing outside firms access to their systems to ensure safety and rational AI development.
Nvidia NVDA.O CEO Jensen Huang dismissed suggestions that AI development should pause — a line he has taken before — arguing instead that increasingly powerful systems are essential for the technology’s progress.
Meta META.O, whose CEO, Mark Zuckerberg, is credited with popularizing the “move fast and break things” ethos, took the opposite view, arguing that each lab should be responsible for setting its own pace, rather than seeking out industry coordination. “Labs face significant liability if their models cause harm, so they have a strong incentive to prevent this,” wrote Zuckerberg in a social media post.
The industry’s self-reflection continued on Wednesday, with Microsoft’s AI chief, Mustafa Suleyman, cautioning that Anthropic’s development of models that imitate human consciousness was ill-advised. "We're all focused on the same aim, which is to try to control a superintelligence," Suleyman told Reuters. "I think that's going to be the greatest challenge that we face in the 21st century."
Still, even as OpenAI appeared to embrace industry calls for a slowdown, there were reports that investor faith had not waned. The company is considering a new funding round that would double its valuation.
That new number: $1.5 trillion.
QQQ•
