OpenAI: Threat actors use us to be efficient, not make new tools

A long-running theme in the use of adversarial AI since the advent of large language models has been the automation and enhancement of well-established hacking methods, rather than the creation of new ones.  

That remains the case for much of OpenAI’s October threat report, which highlights how government agencies and the cybercriminal underground are opting to leverage AI to improve the efficiency or scale of their hacking tools and campaigns instead of reinventing the wheel.

“Repeatedly, and across different types of operations, the threat actors we banned were building AI into their existing workflows, rather than building new workflows around AI,” the report noted.

The majority of this activity still centers on familiar tasks like developing malware, command-and-control infrastructure, crafting more convincing spearphishing emails, and conducting reconnaissance on targeted people, organizations and technologies. 

Still, the latest research from OpenAI’s threat intelligence team does reveal some intriguing data points on how different governments and scammers around the world are attempting to leverage LLM technology in their operations.

One cluster of accounts seemed to focus specifically on several niche subjects known to be particular areas of interest for Chinese intelligence agencies. 

“The threat actors operating these accounts displayed hallmarks consistent with cyber operations conducted to service PRC intelligence requirements: Chinese language use and targeting of Taiwan’s semiconductor sector, U.S. academia and think tanks, and organizations associated with ethnic and political groups critical of the” Chinese government,” wrote authors Ben Nimmo, Kimo Bumanglag, Michael Flossman, Nathaniel Hartley, Lotus Ruan, Jack Stubbs and Albert Zhang.

According to OpenAI, the accounts also share technical overlaps with a publicly known Chinese cyber espionage group.

Perhaps unsatisfied with the American-made product, the accounts also seemed interested in querying ChatGPT with questions about how the same workflows could be established through DeepSeek — an alternative, open-weight Chinese model that may itself have been trained on a version of ChatGPT.

Another cluster of accounts likely tied to North Korea appeared to have taken a modular, factory-like approach to mining ChatGPT for offensive security insight. Each individual account was almost exclusively dedicated to exploring specific use cases, like building Chrome extensions to Safari for Apple App Store publication, configuring Windows Server VPNs, or developing macOS Finder extensions “rather than each account spanning multiple technical areas.”

OpenAI does not make any formal attribution to the North Korean government but notes that its services are blocked in the country and that the behavior of these accounts were “consistent” with the security community’s understanding of North Korean threat actors.

The company also identified other clusters tied to China that heavily used its platform to generate content for social media influence operations pushing pro-China sentiments to countries across the world. Some of the accounts have been loosely associated with a similar Chinese campaign called Spamouflage, though the OpenAI researchers did not make a formal connection.

The activity “shared behavioral traits similar to other China-origin covert influence operations, such as posting hashtags, images or videos disseminated by past operations and used stock images as profile photos or default social media handles, which made them easy to identify,” the researchers noted. 

Another trait the campaign shares with Spamouflage is its seeming ineffectiveness. 

“Most of the posts and social media accounts received minimal or no engagements. Often the only replies to or reposts of a post generated by this network on X and Instagram were by other social media accounts controlled by the operators of this network,” they added. 

OpenAI’s report does not cover Sora 2, its AI video creation tool. The tool’s deepfaking and disinformation capabilities have been the subject of longstanding concern since last year when it was announced, and in the week since its release the invite-only app has already shown a frightening potential for distorting reality.

A rising AI-fueled scam ecosystem and dual use “efficiency”

OpenAI also battles challenges from scammers who seek to use its products to automate or enhance online fraud schemes, ranging from lone actors refining their own personal scams to “scaled and persistent operators likely linked to organized crime groups.”

Most usage is unsurprising: basic research, translating phishing emails, and crafting content for influence campaigns.Yet, OpenAI’s research reveals that both state and non-state actors use AI as a development sandbox for malicious cyber activities and as an administrative tool to streamline their work.

One scam center likely located in Myanmar used ChatGPT “both to generate content for its fraudulent schemes and to conduct day-to-day business tasks,” like organizing schedules, writing internal announcements, assigning desks and living arrangements to workers and managing finances.

Others leveraged the tool in increasingly elaborate ways, like a Cambodian scam center that used it to generate “detailed” biographies for fake companies, executives and employees, then used the model to generate customized social media messages in those characters’ voices to make the scam appear more legitimate. In some cases, the same accounts returned to query ChatGPT on responses they received from target victims, indicating the scheme was somewhat successful.

Researchers also found an interesting dual-use dynamic: in addition to being used by scammers, many users look to ChatGPT for insight about potential scams they have encountered. 

“We have seen evidence of people using ChatGPT to help them identify and avoid online scams millions of times a month; in every scam operation in this report, we have seen the model help people correctly identify the scam and advise them on appropriate safety measures,” the OpenAI researchers claimed, while estimating that the tool is “being used to identify scams up to three times more often than it is being used for scams.”

Because OpenAI claims its model rejected nearly all “outright malicious requests,” in many cases threat intelligence professionals are sifting through clusters and accounts that operate in the “gray zone,” pushing the model to fulfill requests that are dual use in nature and not strictly illegal or against terms of service. For example, a process for improving a tool debugging, cryptography, or browser development can “take on a different significance when repurposed by a threat actor.”

“The activity we observed generally involved making otherwise innocuous requests … and likely utilizing them outside of our platform for malicious purposes,” the authors note.

One example: a group of Russian-speaking cybercriminals attempted to use ChatGPT to develop and refine malware, but when those initial requests were rejected, they pivoted to “eliciting building-block code … which the threat actor then likely assembled into malicious workflows.”

The same actors also prompted the model for obfuscation code, crypter patterns and exfiltration tools that could just as easily be used by cybersecurity defenders, but in this case the threat actors actually posted about their activity on a Russian cybercriminal Telegram.

“These outputs are not inherently malicious, unless used in such a way by a threat actor outside of our platform,” the authors claimed.