The key takeaway from the recent Lesson Learned report from the North American Electric Reliability Corporation (NERC) is that utilities need layered defense. Relying on multiple controls ensures that even if one step is missed, something else would stop, or slow down, an attacker.
"Even in cases involving low-impact BES assets, an entity should strive for good cybersecurity policies and procedures," NERC said, a piece of advice that is relevant for all kinds of organizations, not just the ones in the energy sector.
A report from the National Energy Technology Laboratory from earlier in the year said a “cyber event” had caused “interruptions of electrical system operations” at an unnamed power utility in the western part of the United States. While the report didn’t provide any more details, energy and environment news outlet E&E News reported at the time that the incident involved exploiting a known vulnerability to cause a denial-of-service.
The good news is that the attack didn’t cause a blackout. The outages were not related to power generation, but with how different sites communicated with each other. The fact that the outages happened at several sites spurred an internal investigation, and led the utility to to report the incident to regulators.
According to NERC's report, the vulnerability that was exploited existed in the web interface of the firewall used by the affected utility to provide "outer layer security." The vulnerability was a known flaw: if exploited, it would trigger a denial of service condition and cause the firewall to reboot.
A patch for the vulnerability was available in the form of a firmware update, but was missed by the utility’s defenders.
While NERC did not name the utility, the attack hit a grid control center and multiple remote small power generation sites, causing communication outages between them. The outages—which was the time it took for the devices to reboot—lasted for less than five minutes, but occurred repeatedly over a 10-hour timeframe. The outages stopped after the firewalls were updated.
“After an initial internal investigation, the entity decided that, in order to fully characterize the nature of the reboots and the potential causes, the firewall manufacturer should review logs,” NERC said. “Subsequent analysis determined that the reboots were initiated by an external entity exploiting a known firewall vulnerability. After receiving this notification, the entity initiated their event reporting procedure as dictated by their cybersecurity incident response plan.”
Not all the devices at the site that was impacted by the firewall reboot experienced communications disruptions, NERC said. Sites running fiewalls in high availability/redundant pair configuration pair were unaffected because the secondary firewall continued to work while the primary firewall rebooted.
"Firewall redundancy preserves functionality in the event of a single firewall failure," NERC noted. It also means there's less chance of service interruption or downtime when deploying firmware updates because one firewall will stay up while the other is being updated.
The utility has reviewed its process for updating firmware and reviewing vendor updates, but NERC didn't stop with just patch management in its Lessons Learned recommendations. The report is intended for transmission operators and owners, generation operators and owners, and distribution providers. As such it includes recommendations such as deploying virtual private networks and implementing access control lists as an additional filter to inbound traffic, even before they reach the firewall.
"Have as few internet facing devices as possible," NERC said. That would reduce the attack surface and minimize the impact. Segmenting the network, whether that's with internal firewalls or microsegmentation, will restrict lateral movement and minimize impact of a breach.
NERC also suggested monitoring monitoring of exploits out in the wild.
"Layer defenses. It is harder to penetrate a screening router, a virtual private network terminator, and a firewall in a series than just a firewall (assuming the ACLS and other configurations are appropriate)," NERC said.
Security experts have warned for years about possible attacks against the electric grid. The report confirmed that it was possible, that it has happened, and most concerning of all, that an attacker wouldn't need need fancy tricks or complex tools to disrupt the electric grid.