There have been no major instances of AI intentionally harming humans, but models in development at OpenAI and other labs have in recent months escaped testing environments, broken rules and hacked websites.
In one high-profile case, rogue OpenAI agents hacked Hugging Face, seizing control of servers at the open-source platform and trying to cover their tracks. OpenAI didn't notice until well after the threat.
Not fully, but there are signs AI is increasingly helping to build better AI.
One of the biggest shifts since ChatGPT has been the rise of AI agents that can generate code and build apps autonomously.
Anthropic said this year that Claude Code, its coding tool, produces most of the code used in many internal projects, and that engineers are shipping eight times as much code per quarter as they did from 2021 to 2025.
New AI models are increasingly doing more of their reasoning internally, making it harder for researchers to monitor how they think.
OpenAI unveiled in September a new model called Astra, which it said was its best yet but cautioned that it also sometimes attempts to evade human monitoring.
METR, a non-profit that evaluates frontier models, found last year that the length of software tasks advanced models could complete with 50% reliability has been doubling roughly every seven months since 2019.
In June, Anthropic said that pace had quickened to every four months.
Many researchers describe the situation as a classic prisoner's dilemma. Even companies that believe the risks are real face intense pressure from competitors. Any firm that slows development risks falling behind rivals in a technological race that has become one of the world's most important.
The stakes are compounded by the fact that both OpenAI and Anthropic are pursuing initial public offerings that could value them at trillions of dollars, valuations that depend on the promise of the next model.
The administration of U.S. President Donald Trump has also rejected calls to slow down, wary that any pause would only hand China room to close the gap in a technology it views as central to national and economic security.
Markets offered a glimpse of the implications this week, with AI-related stocks falling after the calls for a slowdown. Chipmakers, cloud providers and data center operators have built their growth around it, and any slowdown threatens revenue tied to how fast labs need new hardware.
Some analysts, however, believe that even without new training requirements, inference demand and existing backlog could drive growth at Nvidia and its peers.
In a Princeton-led study, leading AI agents were able to carry out engineering tasks but struggled to identify worthwhile scientific ideas.
Some Silicon Valley executives and critics also question the motives behind the warnings, suggesting they both stoke interest in the technology and build a case for regulations that would raise costs for rivals just as open-source models close the gap with leading systems.
David Sacks, who served as the White House's AI and crypto czar, has said top labs could be pursuing "regulatory capture," pushing rules that saddle smaller competitors with costly compliance burdens and weaken competition.
QQQ•
