Workado settles with FTC over allegations it inflated its AI detectors’ capabilities 

The Federal Trade Commission thinks AI detectors might be BS.

The agency announced a consent order this week with Workado, an Arizona-based company that makes an AI content detector tool. The order forces the company to  retract its public claims about the tool’s effectiveness and to notify its customers. 

The settlement follows an investigation by the FTC this past year into Workado’s public claims that its AI content detector could determine with near-perfect accuracy whether a piece of text was generated by popular commercial large language chat models from OpenAI, Anthropic, Google and others.

That included claims that the detector “is one of the most trusted and goes deeper than a generic AI detector.” It claims to accurately detect AI-generated content 98% of the time, while at the same time offering a pro version of the software that it claimed could “transform AI text into undetectable AI content.”

But according to an FTC complaint in April, Workado “did not build, train or finetune” the actual AI model behind its product, which was pulled from Hugging Face, an open-source and publicly available AI repository.

That model was only trained on academic content — not Wikipedia, blogs and other sources — and limited to ChatGPT, excluding other commercial models. The developers’ testing data “also showed that the AI Model struggled to identify AI-generated content as AI-generated when evaluating nonacademic content, correctly detecting AI-generated text merely 53.2% of the time,” not 98% as Workado claimed.

“Consumers trusted Workado’s AI Content Detector to help them decipher whether AI was behind a piece of writing, but the product did no better than a coin toss,” said Chris Mufarrige, Director of the FTC’s Bureau of Consumer Protection, in April. 

The FTC settlement specifies that Workado “must not make any representation expressly or by implication” about the effectiveness of its product at detecting AI-generated or altered content “unless the representation is non-misleading.”

In order to do that, Workado must ensure that “at the time such representation is first made, and each time such representation is made thereafter, they possess and rely upon competent and reliable evidence, which when appropriate based on the expertise of professionals in the relevant area must be competent and reliable scientific evidence, that is sufficient in quality, quantity, and timeliness based on standards generally accepted in the relevant fields when considered in light of the entire body of relevant and reliable evidence, to substantiate that the representation is true.”

Essentially, that means every time Workado publicly claims its software can spot signs of AI manipulation, it must repeat its testing process and update the software to keep pace with newer models. As part of the order, the company is required to securely store all test data and related documentation for future review and submit to ongoing government compliance monitoring.

Workado, which did not formally acknowledge wrongdoing as part of the order, must also contact its customers using an FTC-drafted letter to acknowledge it settled charges of false or unsubstantiated advertising claims about the accuracy of its AI content detector.

“We claimed that our AI Content Detector will predict with a 98% accuracy rate whether text was created by AI content generators like ChatGPT, GPT4, Claude, and Bard,” the draft letter states. “The FTC says we didn’t have proof to back up those claims. We’ve stopped making those claims. In the future, we won’t make claims about the accuracy of our AI content detection tools unless we can prove them.”

To be clear, designing a program that can reliably detect AI-generated content models over long periods of time is a challenging, but legitimate, field. Because both deepfakes and deepfake detectors are built on the same underlying LLM technology, their algorithms can learn from each other’s innovations and models can be trained to more effectively find (or evade) each other. This creates a perpetual cat-and-mouse game, where the effectiveness of AI detectors gradually degrades over time unless they’re updated.

Researchers at the Defense Advanced Research Projects Agency (DARPA) have been aware of this problem for years and have worked to design systems that can both accurately identify AI-manipulated content in text, video and audio. They also have designed these systems to be adaptable,  allowing them to evolve as AI technology advances.. While there is a clear need for forensic tools to analyze media for synthetic content, creating solutions that can consistently remain effective will always be a moving target.

But the investigation and settlement with Workado demonstrates that the FTC understands the fluidity of the science behind AI detection, and believes the bar for companies to claim their tools work as intended is higher and requires constant, science-backed vigilance to remain true over time.