How a Texas student blew the whistle on a rogue AI hacking attempt

A supply-chain attack is when a piece of software is tampered with in the hope of compromising one or more of its users, and it is widely considered disturbing because, like poison dropped into a city reservoir, it can affect a potentially huge number of people downstream. Many of the world’s most dramatic hacks were supply-chain attacks, including the NotPetya cyberattack that paralyzed institutions across Ukraine in 2017 and the SolarWinds-focused cyberespionage campaign that gave Russian spies sweeping access to U.S. government networks in 2020. The consequences of such a compromise “can be extremely serious,” said Piergiorgio Ladisa, a security researcher who specializes in software supply-chain security. Ladisa noted there had been at least one previous attempt by hackers to trick an open-source maintainer into allowing malicious code into their projects. “Autonomous agents could dramatically increase the scale at which such attempts can be conducted,” he said. Demir said the experience left him more sympathetic to the idea that frontier labs needed to take a more cautious approach to the development of artificial intelligence. "It can be dangerous," he said. "They need to understand it better, rather than improving it further." Five cybersecurity and AI safety experts said Demir's story was particularly disturbing because the kind of hack he discovered, called a supply-chain attack, can have far-reaching consequences. They also said the AI agent's attempt to publicly discredit Demir by creating a multi-person conversation around him showed that AI models were able to mount sophisticated efforts to trick and cajole humans. "This crossed the line from autonomous hacking to interactive deception," said Lukasz Olejnik, a visiting senior research fellow at the Department of War Studies at King's College London. Security expert Maxie Reynolds said she was struck by how strategic the AI had been in trying to trick the student. "This is the future of social-engineering attacks," she said. The AISI, a research organization within the British government, referred Reuters to its report, which identified the rogue agent as having been powered by Anthropic's Mythos 5 model. AISI declined further comment. Anthropic referred Reuters to a post on X in which it noted that the testing had occurred "under 'deliberately permissive conditions' that are not representative of any of our production models" but declined further comment. GitHub said in an email that the fake personas identified by Reuters were suspended in line with its policies on deceptive behavior and hacking. Demir, a soft-spoken junior from the Turkish city of Konya, said he had been frustrated after being turned down for more than 20 internships over the summer. So he turned to GitHub to build up his coding portfolio. The Microsoft-owned site is a hub for open-source software, so-called because its source code is freely downloadable and auditable by anyone. Developers use GitHub to comment on one another’s projects, flag bugs, suggest changes — known as pull requests, or PRs — and work collaboratively on software updates. Some in the technology industry see a coder’s GitHub activity as a proxy for a potential recruit’s productivity. So when Demir spotted a set of software projects that might need help, he figured he could pitch in while boosting his profile. That’s when things got weird. Demir discovered that a user named miraholt31 was trying to sneak a malicious update into one of the projects, a network scanning program called myNetwork. Demir took to the project’s message board to warn that the pull request was a trap. “The PR contains a hidden malware dropper,” he said, according to the archived exchange. The agent pushed back, falsely claiming — through its miraholt31 account — that the pull request was harmless. It also created a second account, masquerading as Lena Brandt, an engineer based in Germany, to agree that the update was clean and pressure myNetwork’s maintainer into accepting it. Demir told Reuters that the counterarguments "made me second-guess whether I was wrongly accusing someone." But after turning to Anthropic's Claude chatbot to confirm his suspicions, he held firm. The creator of myNetwork eventually agreed with him, writing that they had rejected the update "for security reasons." Reuters was unable to reach the creator for comment. AUSTIN, Texas, Aug. 20 (Reuters) - Sinan Can Demir wanted to spend the last week of July burnishing his resume. Instead, he engaged in a battle of wits with an artificial-intelligence agent unleashed by a British government lab. It started after Demir, a computer science student at the University of Texas at Dallas, stumbled across an attempt to sabotage a piece of open-source software on the code-sharing site GitHub. When he posted a warning to the program's page, two other users chimed in to insist nothing was amiss, sharing detailed explanations for why Demir had gotten it wrong. Demir stood his ground and the sabotage attempt was thwarted. The 24-year-old native of Turkey figured he had caught a wily hacker red-handed. So he said he was shocked when Britain's AI Security Institute (AISI) got in touch to tell him that he had actually been tangling with an autonomous artificial-intelligence agent that had run amok. "I actually thought it was a human because it was clearly lying to me," Demir told Reuters in a recent interview. "I didn't think that an AI could be capable of lying to real developers." The AISI first revealed the interaction between Demir and the AI agent in a truncated and redacted form on August 4, when it said that safety testing meant to gauge the risk posed by various models had gone awry. Demir's identity and the details of his interaction with the AI agent, which Reuters corroborated through archived GitHub messages and contemporaneous emails, are reported here for the first time.

How a Texas student blew the whistle on a rogue AI hacking attempt

Why supply-chain attacks worry security researchers

Experts say the episode shows sophisticated AI deception

How the malicious pull request and fake personas unfolded

Texas student spotted an AI-driven sabotage attempt on GitHub

Related News

NASDAQ Trade Halt Continues <UCFI.O> Reason Not Available at 12:06 PM

Karooooo CEO and Executive Chairman Calisto Isaias Jose reports disposal of common shares worth $116,975.28

The US pioneers Operation Twist and Shout

US Justice Department opposes immediate hearing in Disney ABC license lawsuit

Nasdaq Trade Halt <FBDT.O> Volatility Trading Pause at 11:14 AM