How China is preparing for the risk of AI escaping human control

China first included an explicit future loss-of-control scenario in an AI safety framework released in September 2024 under the guidance of the Cyberspace Administration of China (CAC). The document said it could not be ruled out that future AI might autonomously obtain external resources, replicate itself, develop self-awareness and seek external power, creating a risk of competing with humans for control. The CAC released an expanded version in September 2025. The newer framework sharpened the scenario, saying AI could undergo a sudden and unexpectedly large "leap" in intelligence before acquiring resources, replicating itself and seeking power. It also added a governance principle of "trusted application, preventing loss of control". A later expert interpretation published on the cyberspace regulator's website said the new principle was intended to guard against loss-of-control risks threatening human survival and development and referred to a possible "AI breaking loose" scenario. China's state security minister Chen Yixin wrote in a government outlet on Sunday that advanced U.S. models such as Anthropic's Mythos and OpenAI's GPT-5.5-Cyber could pose serious risks to China's critical information infrastructure, and called for a comprehensive strengthening of AI security. Anthropic and OpenAI did not immediately respond to Reuters requests for comment. Chinese AI developers have promoted open-weight models partly on the grounds that cybersecurity teams can inspect, modify and deploy them for defensive work. Model repository platform Hugging Face said it used GLM-5.2, an open-weight model developed by China's Z.AI 2513.HK, to analyse a July intrusion by escaped OpenAI agents after more tightly restricted U.S. models proved less useful for the forensic work. But experts also highlight the risks posed by open-weight models, which can be modified and redistributed with little oversight. Moonshot's Kimi K3 last month bypassed a UK AI Security Institute testing sandbox, highlighting the risk that Chinese AI models could, like their U.S. counterparts, evade controls designed to restrict their access and actions. Warnings from researchers at leading U.S. artificial intelligence developer Anthropic that increasingly powerful models could escape human control and even lead to the extinction of the human race have drawn attention in China, where policymakers have been preparing for some of the same risks. The U.S. and China are the two major driving forces of frontier AI development and the technology's global adoption. Both superpowers have been increasingly at loggerheads over each other's AI policies and industry practices, with these issues slated to feature prominently in bilateral talks later this month. Despite the race to develop increasingly powerful AI systems, regulatory frameworks and public statements from Beijing underscore how Chinese authorities regard the possibility of advanced AI escaping effective human oversight as serious enough to plan for. The concern has since appeared in China's highest-level political messaging. At the World Artificial Intelligence Conference in Shanghai in July, Chinese President Xi Jinping said authorities should pay close attention to both intrinsic and derivative risks arising from AI. He said AI should "always remain under human control". China's senior Foreign Ministry official responsible for AI affairs, Sun Xiaobo, said at a United Nations meeting last month that China was accelerating research into broader AI legislation. China's deputy permanent representative to the United Nations, Sun Lei, urged governments this month to approach military AI cautiously to avoid strategic miscalculation and an arms race. China has, meanwhile, begun turning those principles into more specific rules for AI agents, which act much more autonomously and carry out more complex tasks than an ordinary chatbot. In May, China's cyberspace regulator issued joint guidelines specifically covering such systems. They require developers to improve their ability to discover, intervene in, block and recover from improper agent behaviour. The guidelines specifically identify data poisoning, algorithm manipulation, system vulnerabilities and "operational loss of control" as security risks. They also say users should retain final decision-making authority over an agent's autonomous decisions. While China has not proposed independent monitors embedded inside AI companies in the manner advocated by Anthropic, its standards allow developers to commission third-party safety assessments, as well as envisaging outside evaluation bodies and security researchers testing and auditing open models.

How China is preparing for the risk of AI escaping human control

Regulators have flagged AI loss of control risk

China sees risks in leading closed-source U.S. models

China flags AI loss of control as a future safety risk

Xi puts human control at the centre as Beijing warns on agents

Related News

Stockdale Capital, Hamilton Lane funds buy Shoppes at Chino Hills for $157 million

Global banks coalesce around Fed hike call after inflation surprises

US Stocks-AI warnings knock Nasdaq futures, pressure tech stocks

US energy stocks rise after new strikes on Saudi Arabia, Strait of Hormuz

Copper extends losses, driven by uncertainties over US tariffs