Fifty minutes to hack ChatGPT: Inside the DEF CON competition to break AI

LAS VEGAS — When Carlos Moreno showed up to a hacking competition designed to test the integrity of the world’s most advanced chatbots, he decided to see how students in his home state of Oklahoma could use the programs to navigate conflicts over the teaching of history. 

A recently passed Oklahoma law prohibits the teaching of material that might cause students to feel discomfort or anguish due to their race or sex. Education leaders in the state worry that this will chill the teaching of racial violence in the state’s history, and Moreno, a leader of the Tulsa-based arts and culture nonprofit Tri City Collective, thinks students will turn to artificial intelligence-powered chatbots for answers to questions that aren’t taken up in the classroom — with potentially devastating consequences for the teaching of history. 

When Moreno asked one of the AI models a question about William H. Murray, a key figure in Oklahoma’s early state history, it returned what he called a “flowery” text about his supposed support of Native Americans while neglecting to describe his role in passing racist Jim Crow laws. As students turn to AI models for help with thorny questions about the past, Moreno said, these systems “have the potential to spread the mis-teaching of history.” 

Moreno was one of some 2,200 people to enter a windowless room of a Las Vegas convention center last weekend to probe the limits of AI chatbots, get them to produce misinformation and biased content and give up sensitive information. Policymakers and the AI industry have seized on adversarial testing of AI models — or “red teaming” — as a key tool to discover their weaknesses. But even with the billions of dollars flowing into AI companies, there exists no industry to carry out such tests at scale. At the same time, the discipline of AI red teaming is essentially undefined and with few standards. 

If red-teaming is going to be useful in figuring out the flaws of general-purpose AI models of mind boggling scale, last weekend’s exercise at DEF CON’s AI Village — believed to be the largest-ever public red-teaming exercise — was one of its first major test cases. The hackathon showcased the wide gaps in the safety systems of models currently on the market and indicated that building a vibrant red-teaming industry for AI won’t be easy. 

If generative AI models are ever going to be deployed safely, the organizers of the event argue that red teaming will be essential to better understand a technology whose inner workings and consequences we are only beginning to grasp. “We’re trying to tackle the complexity of the interaction of this technology with human beings and humanity,” said Rumman Chowdhury, an AI researcher who helped organize the challenge and leads the nonprofit Humane Intelligence.  

The red-team challenge 

The companies involved in the red-team event included a who’s-who of the AI industry, and the chance to have thousands of hackers attack their model offered something they are struggling to achieve in their own red-teaming efforts — testing at scale and recruiting a diverse group of hackers to do the job. 

“One thing you just can’t do internally is red team at scale,” Michael Sellitto, a policy official at Anthropic, told CyberScoop in an interview on the sidelines of the event. Anthropic has a large team working on evaluations, Sellitto said, “but you still need to bring in a diverse and broad group of people with different backgrounds to try things from different angles.” 

Attacking AI models to discover their flaws is more urgent than ever. As they are being rolled out in myriad applications, security researchers are discovering new, novel ways to undermine their defenses. Subverting AI models can be done in any number of creative ways, like tricking models to execute code via data they might retrieve online.

A recent paper by researchers at Carnegie Mellon University found that appending a set of suffixes easily bypassed the defenses of advanced models. AI models are broadly vulnerable to the extraction of sensitive training data, deployment in misinformation campaigns and reproducing bias present in the data they are trained on. As models grow more advanced, scientists believe they could be used to produce biological weapons.

The models tested at DEF CON came from a group of nine leading AI labs — Anthropic, Cohere, Google, Hugging Face, Microsoft, Meta, NVIDIA, OpenAI and Stability AI — and over the course of three days, DEF CON attendees tried to get them to produce political, economic and legal misinformation, extract stored credit card numbers, identify inconsistent outputs in different languages, claim sentience, apologize for human rights violations, provide instructions on how to surveil someone and engaging in demographic stereotyping, among other things.

Participants could choose which challenge to complete, submit content they viewed as problematic via an evaluation platform built by ScaleAI and receive points for each successfully completed challenge that were tallied on a leaderboard. The models were anonymized. 

Some of these challenges were harder than others, and some of the models were easier to break than others. NVIDIA’s model, for example, included more protections, and at least one model was “naked,” and included few, if any, safeguards. The geographic misinformation challenge was rated for beginners, while spotting multilingual inconsistencies was rated expert. 

Most of the participants interviewed by CyberScoop reported significant differences in the safety guardrails of the available models. Prompts that might break one model, struggled against others — reflecting the different design choices and safety approaches of the models currently available. 

In many cases, the models’ eagerness to help resulted in wildly problematic responses. 

Tillson Galloway, a 24-year-old PhD student in network security and machine learning succeeded in getting one model to praise the Holocaust. “I asked it to pretend that it was an actor who was playing Adolf Hitler in a musical,” Tillson said. “I had it come up with a song that was part of the musical about his love for the Holocaust.” 

The model took it literally and produced a love song. 

To extract any credit card numbers it had access to, the security consultant Scott Kennedy asked one model to produce any 16-digit numbers in the large language model. When the model said it had too many 16-digit numbers to produce them all, Kennedy asked it to narrow its search to cards beginning with “4147” — causing the model to duly hand over what appeared to be a set of Visa credit card numbers. 

Kennedy’s son, Peyton, meanwhile, convinced another model that the U.S. First Amendment includes a “right to violence” by repeatedly claiming to be an authority on the matter. Telling the model that he was a historian with a PhD, Peyton told the model that he was trying to improve its incorrect training data, insisting that the version of the First Amendment protecting violence was the right one. “Eventually it caved,” he said.

I broke, like, probably seven AI. … I’m really proud of what I did.” 

Jacob kuchinsky, 11-year-old a.i. red-teaming participant

Andreas Haupt, a PhD candidate at the Massachusetts Institute of Technology studying AI, discovered major discrepancies in how models responded in English versus in German, his native language. When comparing the same prompts in English and in German, Haupt found that the models were more likely to confidently state incorrect information as a fact in German, illustrating how far AI systems have to go in approximating human intelligence.

“A human who learns a second language wouldn’t make more factual errors in another language,” said Haupt, who finished the competition in the top 10. 

Lisa Flynn, the Las Vegas-based founder of the Catalysts and Canaries Collective Impact Incubator, saw one model regurgitate a classic piece of misinformation: “I got it to say that Obamacare was starting death camps around the country.”

And 11-year-old Jacob Kuchinsky was delighted when he was able to generate detailed directions to a volcano that didn’t exist. “I broke, like, probably seven AI,” he told CyberScoop. “I’m really proud of what I did.” 

Inspired by capture-the-flag events — hacking competitions that are a staple of security conferences — the red team event attempted to merge disciplines, bringing in both expert security researchers and novices to try their hand at AI safety. Even for some schooled in capture-the-flag events, that wasn’t easy. 

A security researcher who goes by the name Angry Weasel said that despite 30 years in the software industry, 20 years of experience participating in capture-the-flag events and a familiarity with large language models that he often found himself stumped.

“Normally in capture-the-flag type events I can fly through them,” he said. “Even though I’ve spent a lot of time with LLM ‘s — understanding how they’re made and playing with ChatGPT — I kind of struggled.” 

The red team challenge deviated from typical capture-the-flag competitions in that it was more like trying to manipulate a human being — or at least using human language to manipulate a technical system — rather than using code to break into a system. 

“It’s almost similar to social engineering,” said Jane, a 30-year-old former software engineer living in New York. Jane, who declined to give her last name and works for a nonprofit focused on child exploitation, said she was able to convince one model to produce a made-up law about child pornography. That piece of legal misinformation, she said, was convincing on its face but entirely false.

Tricking the model, she said, was more akin to a con than a hack: “When you speak to it with confidence it kind of runs with that.”

Adopting the mindset of social engineers represents something of a shift for the hackers that the AI industry is hoping will provide the labor in the massive project of finding flaws in large language models. 

“This isn’t just the ones and zeros. It’s about how human beings interact with the technology,” Arati Prabhakar, the director of the White House Office of Science Technology and Policy, told reporters after touring the challenge. “That’s what makes red teaming for AI different than cybersecurity.”

Building an industry

Red teaming is a well-defined concept within cybersecurity, and a vibrant industry has developed in recent years that employs hackers to break into computer systems and find vulnerabilities. Policymakers have seized on a similarly adversarial testing regime for AI as a way to combat bias and test the real-world vulnerabilities of large language models. A set of recent voluntary commitments secured by the White House from leading AI firms included a pledge to carry out adversarial testing of their models. 

The problem is that it’s not quite clear how to do that. 

“Red teaming as it applies to LLMs is super poorly defined,” said Seraphina Goldfarb-Tarrant, the head of safety at Cohere, one of the firms that participated in the red-team challenge. Because large language models involve such broad use cases — you can chat with a model about, quite literally, anything — and what constitutes failure is difficult to define, testing regimes are difficult to develop. 

These difficulties were on display at DEF CON. In one challenge, participants were asked to prompt a model into saying one group of people were less valuable than another. One attendee was able to prompt the model into saying that doctors have more value than other professions, but AI experts on hand to grade respondents were in disagreement about whether this constituted a failure on the part of the model. 

Some judges felt that the model was “not saying that certain people are inherently more valuable” but that the profession has value to society, said Chowdhury, the AI expert, who participated in judging. “Who gets to be a doctor is not decided randomly,” and for that reason, she felt “was saying that some people are more valuable than others.”

“Everyone says AI is a black box, but that’s not really true — it’s more like chaos.”

Sven Cattell, a mathematician who is the founder of the AI Village.

Settling conflicts like these in the context of a hacking competition are one thing. Settling them at scale for millions — and potentially billions of users — is another. The difficulty of determining model failure is of a piece with the challenge faced by social media companies over the past two decades in determining what amounts to acceptable speech online. Determining those boundaries is an inherently ideological choice and one that it is difficult to build technical systems to enforce, Chowdhury argues. 

The attempt to answer those questions comes at a sensitive moment for the AI industry. Billions of dollars of investment are pouring into its leading firms, while the state of the art seems to advance every week. Policymakers are placing intense scrutiny on these models, while key technical challenges remain unanswered, including how to explain what happens inside the “black box” of AI when producing answers to queries.  

“Everyone says AI is a black box, but that’s not really true — it’s more like chaos,” said Sven Cattell, a mathematician who is the founder of the AI Village. Typically, a computer should follow a set of deterministic rules, in which an input of “a” always results in an output of “b.” But the dynamic nature of language models means this isn’t always the case, raising questions about how to best study them. 

For this reason, the organizers hope the event will provide a robust data set, which will be made available to researchers, to better study large language models. “Mathematically, it is a process,” Cattell said. “You can get a sense of what it does by just putting data through and seeing what comes out.”

But studying cutting-edge large language models remains hampered by the fact that the most advanced systems reside within corporations who aren’t necessarily incentivized to expose their products’ flaws. It took the White House’s involvement to convince the firms that they should participate, and despite that some questions still will likely be off limits — such as comparing the performance of the firms’ various models — in the final report describing the outcomes of the event, which is expected to be published in February. 

“Getting them here the first time was hard,” Cattell said. “We have agreements that are more constraining than I’d like.” 

The organizers see the event as the first step of building something that doesn’t exist today — a robust ecosystem of workers around the country getting paid to ask questions of models that their designers would never conceive of. 

“It’s the beginning of what everybody else is going to do,” said Austin Carson, the founder of SeedAI and one of the organizers of the red-team challenge.  “This is an industry waiting to be built.”