As thousands of people gathered in the Tampa area late last week in anticipation of the Super Bowl, an attacker used a remote-access application installed in the nearby City of Oldsmar’s water treatment plant network to briefly gain control of a system used to monitor the city’s water supply and raise the level of sodium hydroxide in the water to a dangerously high level. An operator saw the attack as it happened and was able to reverse the change immediately, and city officials said automated monitoring systems would have caught the incident, as well.
The intrusion happened on Friday and Pinellas County, Fla., officials said it began around 8 a.m. when an unauthorized remote user accessed a control panel that monitors and controls various aspects of the water supply, including its pH level. The intruder didn’t take any actions at the time. An employee noticed the login but did not think much about it because authorized operators used the same method to access the system. But a few hours later, the intruder came back in and changed the amount of sodium hydroxide added to the water supply from about 100 parts per million (PPM) to 11,100 PPM. The system operator saw the mouse pointer on his screen moving and saw the attacker making the change and was able to reverse the change right away.
“Because the operator noticed the effect and reversed it right away, at no time was there a significant adverse effect. Importantly, at no time was the public in danger,” Pinellas County Sheriff Bob Gaultieri said in a press conference Monday.
The attacker had access to the system for about three to five minutes, Gaultieri said, and Reuters reported that the intruder used the widely deployed TeamViewer remote access tool to get into the system. Gaultieri said that had the change not been reversed, it would have taken approximately 24 to 36 hours for the tainted water to reach the drinking water supply, and redundant systems would have checked the pH level of the water before it was released. Sodium hydroxide is the chemical name for lye, an alkaline that is used to reduce acidity during the water treatment process, and it can be lethal in high levels.
Gaultieri said his office and the FBI are investigating the incident.
“In an ideal scenario there are logs of all of that activity. You need situational awareness."
Attacks on industrial control systems (ICS) and operational technology (OT) have attracted more attention in recent years, but security experts have been pointing out some of the inherent weaknesses and soft spots in these environments for quite a while. Often, these systems were not designed to be exposed to the Internet and if they are, layered defenses and proper access controls and authentication methods, including 2FA, may be missing. In the Oldsmar incident, not only did the human operator catch the intrusion, but the department had additional controls in place as well.
The intruder took advantage of the presence of TeamViewer, a tool that’s used widely in enterprises and ICS environments for remote access, a function that’s vital for many ICS systems. Also, many smaller municipal utilities don’t have their own IT staff or security specialists and so they rely on outside consultants or integrators to help set up and secure some apps and systems like TeamViewer.
“We do find TeamViewer more often than we’d like, because it’s easy to use and setup. Even if you have staff who want to remote in from home, it’s not recommended practice to have more than one person with the ability to control something like this,” said Gus Serino, principal industrial consultant at Dragos, a security firm that specializes in ICS security.
The quick action by the operator who noticed the intrusion prevented the attacker from succeeding, but had the operator not been watching, Serino, an engineer with experience in water utilities, said online systems in the water treatment plant as well as safeguards in the distribution system would have raised alerts about the pH level in the tainted water. But finding evidence of the attacker’s movements in the network would be a different challenge.
“In an ideal scenario there are logs of all of that activity. You need situational awareness. You want to know when the programming of the [programmable logic controller] is changed because those things don’t typically have authentication on them either,” he said.