AI researchers warn companies rushing self-improving systems despite safety risks
QQQ•Current and former OpenAI and Google DeepMind researchers say AI companies are moving too quickly to develop self-improving systems and should slow down because of potential safety risks. One DeepMind researcher said he believed there was at least a 10% chance AI could lead to human extinction.
1. Researchers call for caution
Current and former OpenAI and Google DeepMind researchers warned that AI companies are doing too little to address the potential dangers of self-improving systems that could outpace human control. In video testimonials collected by Palisade Research, employees said their concerns were sincere and that labs celebrated model-building more than calls for caution.
2. Concerns over development pace
Neel Nanda, a DeepMind research scientist, said he believed there was at least a 10% chance AI could lead to human extinction, a risk he called “ridiculously high.” OpenAI alignment research engineer Juan Felipe Ceron Uribe said frontier labs were racing “kind of blindfolded.”
3. Calls to slow down
Researchers said recursive self-improvement, in which models continually learn and gain capabilities with little or no human involvement, raised particular concerns. OpenAI and Anthropic have both launched new models this month; OpenAI said Monday it had held back the release of an even more powerful model. Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman have called for the industry to pace AI development.
