It was the kind of doomsday scenario cybersecurity experts had been warning about for years: hackers infiltrate a small water utility and try to poison the local population. And that’s exactly what appeared to happen in February 2021 in Oldsmar, Florida.
News of hackers remotely tampering with levels of lye at the local water treatment facility alarmed officials, shocked the public and has been served as a siren call for the need to safeguard the most sensitive U.S. networks from malicious hackers attempting cause serious physical harm and even death. In the years since Oldsmar authorities first announced the incident, officials in Washington also have regularly pointed to the case as exhibit No. 1 for more cyber investments — and regulations — for U.S. critical infrastructure.
But two years later, there’s still little evidence pointing to exactly what happened inside the plant, how a hacker could have gained access to internal systems or who may have even carried out the alleged attack. Now, new details suggest that the incident may not have been the work of an outside hacker at all. In a statement to CyberScoop, the FBI said that “through the course of the investigation the FBI was not able to confirm that this incident was initiated by a targeted cyber intrusion of Oldsmar.”
The rare comment from the bureau about the investigation resulted from questions about comments from former Oldsmar city manager Al Braithwaite. At a recent virtual conference for public administrators, he made the surprising claim that the incident was a “non-event” spurred by an overzealous employee.
“The FBI concluded there was nothing, no evidence of any access from the outside, and that it was likely the same employee that was purported to be a hero for catching it, was actually banging on his keyboard,” Braithwaite said, according to GCN which first reported the statement. Braithwaite acknowledged but did not respond to further request for comment.
At the very least, the FBI statement and Braithwaite’s comments tell a much different story from how the Oldsmar event was initially characterized by local law enforcement in 2021 and in subsequent articles. “This is somebody that is trying, at least it appears on the surface, to do something bad … It’s a bad actor,” said Bob Gualtieri, the sheriff of Pinellas County, Florida, at the time according to CyberScoop.
In addition to CyberScoop, the event was covered by just about every major outlet with headlines running in The New York Times (‘Dangerous Stuff’: Hackers Tried to Poison Water Supply of Florida Town) and CNN (“Someone tried to poison a Florida city by hacking into the water treatment system, sheriff says”) making it one of the most high-profile cybersecurity events in the U.S. in recent years.
In the years following the Oldsmar news, it was rare to have a congressional hearing on cybersecurity without someone mentioning it. At a budget hearing last April, Cybersecurity and Infrastructure Security Agency Director Jen Easterly used the Oldsmar event to underscore the importance of a $1 billion grant program for states and localities. “Water entities that, frankly, are very target rich — as we saw with Oldsmar in February of 2021 — but resource poor, and so being able to provide grant money to help them raise their cybersecurity baseline, I think, is really important.”
The Oldsmar incident combined with the ransomware attack targeting Colonial Pipeline served to accelerate U.S. officials’ interest in improving critical cybersecurity practices. In March, the Biden administration announced new cybersecurity regulations for the water sector. In a call announcing new cyber mandates, Environmental Protection Agency Assistant Administrator Radhika Fox referenced the Oldsmar incident.
Asked about the incident two years later, the EPA said that the agency collaborates with the Department of Homeland Security, National Security Council and others “to share information with the water sector about found vulnerabilities and countermeasures. EPA is not directly involved in the Oldsmar incident.”
What is known about the alleged digital intrusion at Oldsmar is that on the morning of Feb. 5, 2021, an employee of the water treatment facility reportedly saw someone around 8 a.m. remotely log into the system. The employee apparently dismissed the incident, however, as remote access was not unusual from other employees.
Roughly five hours later, at 1:30 p.m., according to an Idaho National Labs report, an employee watched as someone with access to the human machine interface raised the levels of sodium hydroxide — a chemical used in water treatment to control acidity — to dangerous levels from 100 parts per million to 11,000 ppm. The employee saw the change and was able to change the lye back to safe levels.
The following Monday, Sheriff Gualtieri, Braithwaite and Oldsmar Mayor Eric Seidel held a press conference revealing the news while also noting that multiple safety systems would have prevented any real harm. The FBI, Secret Service and the Pinellas County Sheriff Office were part of the incident response.
Later, a joint advisory from CISA, the EPA and Multi-State Information Security and Analysis Center noted that the plant still used Windows 7, which Microsoft ended support for the year before. Additionally, a data leak containing the email addresses and passwords with two domains belonging to Oldsmar surfaced days before the breach occurred, CyberNews reported.
Early reports, including statements from the Sheriff Gualtieri, said that the remote access tool TeamViewer could have been the initial access point, but this was never confirmed. Other theories ranged from an inside job from a disgruntled employee to Iranian hackers selling access to a water treatment plant in Florida.
But even if the event turns out not to be the work of an outside malicious hacker, the threat to water treatment facilities is still very real, said Jennifer Lyn Walker, director of infrastructure cyber defense at the Water Information Sharing and Analysis Center. Furthermore, she said, the incident helped give the attention needed to kickstart a larger conversation about securing the water and wastewater systems, particularly for smaller utilities. “We’re talking about human lives potentially being at risk.”
“While some of the incident details may have changed,” she said “other findings, other vulnerabilities identified through the investigation are still representative of gaps across the sector and other critical infrastructure and smaller organizations.”
The Water-ISAC did initially raise the possibility that Oldsmar may not have been hacked. Early analysis from the information clearinghouse brought up the possibility that the incident could be “human error” and wrote that it’s worth “at least considering this could have been an authorized connection with an intentional change to an unintentional value.”
“Is it out of the realm of possibility that a level from ‘100’ to ‘110’ or ‘111’ is a plausible change and the trailing zeros were erroneously not deleted, thus giving the impression of an ‘attack,'” the Water-ISAC wrote.
The Secret Service, which was a part of the original incident response, did not respond to multiple requests for comment. CISA referred questions to the FBI. Oldsmar’s current city manager acknowledged a request but did not respond in time for publication.
Deputy Dave Brenn, spokesperson for the Pinellas County Sheriff Office, said in an email that “the case is still open and no further information will be disclosed.”