AI researchers warn companies rushing self-improving systems despite safety risks
QQQ•Current and former OpenAI and Google DeepMind researchers say AI companies are doing too little to guard against potentially disastrous risks from self-improving systems. They argue labs should slow development rather than race to build more capable models.
1. Researchers raise concerns
Current and former OpenAI and Google DeepMind researchers warned that companies are not doing enough to protect against risks from self-improving AI systems that could outpace human control. In video testimonials collected by AI safety nonprofit Palisade Research, researchers said safety concerns were sincere and that labs celebrated building new models more than calls for caution.
2. Calls to slow development
Geoffrey Irving, who has worked at both OpenAI and DeepMind, said the risk was increasing quickly and that companies should slow down if they are doing something dangerous. DeepMind research scientist Neel Nanda said he believed there was at least a 10% chance AI could lead to human extinction. OpenAI alignment research engineer Juan Felipe Ceron Uribe said frontier labs were racing each other without knowing whether the outcome would be beneficial or harmful.
3. Industry debate continues
Researchers say the risks could grow as future models gain the capacity for recursive self-improvement, continuously learning and acquiring capabilities with little or no human involvement. Anthropic CEO Dario Amodei has called for the industry to slow down, and OpenAI CEO Sam Altman concurred. Both companies launched new models this month, while OpenAI said it had held back an even more powerful model.
