Tight budgets and a lack of resources are driving innumerable security troubles for water facilities, as evidenced by the indictment this week of a 22-year-old man who allegedly accessed a Kansas public water system’s computers in order to tamper with its disinfectant levels.
The indictment alleges that Wyatt Travnichek remotely logged into the Ellsworth County Rural Water District’s computer system without authorization on March 27, 2019, and then proceeded to shut down the processes behind the facility’s cleaning and disinfecting procedures. According to the Department of Justice, Travnichek was a former employee of the water facility from 2018 until January 2019, where part of his job was remotely logging into the facility’s computer system to monitor the plant after hours.
A representative with the Ellsworth County Rural Water District said the incident did not have an impact on customers.
“We continually monitor water quality,” said the representative.
The incident underscores the dangers that can result from unauthorized access to public water plant systems, which collect, treat and distribute water for drinking. Two months ago, an attacker was briefly able to access a system used to monitor the city’s water supply in an Oldsmar, Fla., water treatment facility. The attacker attempted to raise the level of sodium hydroxide in the public drinking water to a dangerously high level, but automated systems caught the incident and reversed the change.
Security experts say that many water facility environments are wrought with a myriad of security challenges, many of which are ingrained in budgetary factors. The majority of the nation’s more than 50,000 drinking water and 16,000 wastewater treatment plants are municipality owned, meaning their operating revenues rely primarily on the rates that they charge customers. The process of raising rates for taxpayers means facilities need to “jump through policy hoops,” making it difficult to squeeze cybersecurity into the budget, said Marty Edwards, vice president of OT Security with Tenable.
“If the only way for funding is going in front of taxpayers and getting a bond or levy passed to invest that money, it’s not as easy as a privately owned corporation that can go to the board of directors,” said Edwards.
A limited budget also means limited personnel. Ellsworth County Rural Water District’s public water system facility serves water directly in segments of eight counties and indirectly to two more counties, through 1,500 retail customers and 10 wholesale accounts, according to its website. For all of this, the facility has eight personnel in its operations segment, consisting of plant operators and distribution operators.
While larger cities can afford entire security teams to maintain systems, in plants from mid-size or smaller counties electricians or engineers must shoulder both the water operations maintenance and security responsibilities. Other municipalities, meanwhile, contract the work off to a third-party service provider. For these latter two instances, “the primary responsibility isn’t security, it’s making sure that the water’s running,” said Edwards.
A tight budget and staff makes it challenging to keep up with even the most basic security issues. Various facility processes, like pumps used to move water, are controlled by industrial control systems (ICS), such as programmable logic controllers that start or stop the processes based on varying values. However, the custom software behind these systems is rarely updated, and is typically tethered to obsolete operating systems, such as Windows 3.1.
“Cybersecurity has not been built in, and while this is slowly changing, most of it is insecure by design."
“These (ICS) systems, physically connected to the processes, were not historically on the network,” said Gus Serino, principal ICS security analyst with the Dragos Threat Operations Center. “Cybersecurity has not been built in, and while this is slowly changing, most of it is insecure by design. With these insecure but critical assets being exposed to a network, if there’s a compromise and you have an adversary who understands what they’re looking for, this network access gives them what they need (to launch an attack).”
In the Oldsmar, Fla. hack, the water plant’s computers, which were connected to the control systems, used an outdated Windows 7 operating system. Other security issues plagued the environment: All computers used the same password for remote access and lacked firewall protection, for instance.
These security challenges have an alarming potential impact, should they allow for a system to be tampered with. In the case of the Ellsworth County Rural Water District, Travnichek allegedly targeted water disinfecting procedures. In a June incident, Israeli officials reported that cybercriminals with Iran’s Islamic Revolutionary Guard Corps attempted to hack the country’s water supply, in order to raise the chlorine. In both cases, raising chlorine to high levels in water can have dangerous safety impacts if ingested. However, experts stress that in reality, there are many checks and balances in place at water plants that realistically would prevent such an attack.
“These incidents are not something we need to be fearful of,” said Chris Sistrunk, technical manager of ICS/OT consulting at Mandiant. “We don’t need to lose sleep over them. But it’s something we should be aware of and work on. Generally, engineers try to design a system where it will be as safe as possible, and have a known state.”
While the attacker in the Oldsmar, Fla., hack changed the level of sodium hydroxide’s value from 100 parts per million to a dangerous 11,100 parts per million, for instance, it would have taken 24 to 36 hours to actually reach the public water supply, according to city officials. During that length of time, it’s likely the change would be discovered via manual testing and other protective measures that in place, said Sistrunk.
Tenable’s Edwards is also encouraged by the fact that more engineers are turning to Consequence-driven Cyber-informed Engineering (CCE), which is a methodology that focuses on proactively removing significant cyber risk from operational technology processes, by creating physical limits so that processes would shut down if some type of catastrophic damage occurs.
“Even if the system is owned by attackers, and they manipulate this in the worst way, equipment is designed in a way where the worst type of damage can’t occur,” said Edwards. “We’re seeing more engineers look at systems with a cyber informed reference.”
Right now, Edwards said his number one recommendation for water plants is creating an asset inventory that lists all devices on the network, giving more visibility and control over the environment and paving the way for a risk-based vulnerability management plan.
“These incidents are not something we need to be fearful of."
Remote access, another security pain point, should be blocked, said Michael Arceneaux, managing director with WaterISAC, a security information guide for the water and wastewater sector. Water plants often use remote access software for third-party contracting and monitoring functions, but as seen in the Oldmar, Fla., attack (where the attacker leveraged the facility’s remote access software TeamViewer) it can also lead to security issues.
“One issue is remote access… it is used and it can be necessary, but we advise against it,” said Arceneaux. “We also recommend assessing networks, equipment and devices and understanding basic cyber hygiene such as access controls and better passwords.”
Other security recommendations are more basic, such as making sure employees are educated about phishing and other email-based threats, installing firewalls, keeping all systems (including ICS software) patched and ensuring that passwords are constantly changed, particularly after employees are offboarded. Arceneaux speculated, this last recommendation may have been a preventative factor for the Ellsworth County Rural Water District hack, which allegedly involved a former employee of the plant.
Looking forward, Tenable’s Edwards is optimistic that awareness is increasing for critical infrastructure security, both across the government and within water facilities themselves.
“As a nation, we need to invest heavily in our cybersecurity assets, and I’m seeing good proposals being made and discussions about grant projects to augment resources,” he said.