For the thousands of SolarWinds customers who may have installed a trojaned update planted by attackers earlier this year, the next few days and weeks will be tense and stressful as the incident response teams work to determine what, if any, damage has been done. But, because of the way the intrusion happened and the way the SolarWinds platform works, it may be much longer before many organizations know the full scope of the problem.
On Sunday, FireEye and Microsoft published details of an operation in which an attacker was able to access the internal network of SolarWinds, an enterprise IT monitoring software maker, get to a build server, and plant a malicious update file on the server. When that update made its way onto customers’ networks, the attackers then had a mechanism to install a backdoor, giving them access to the SolarWinds Orion deployments on those networks, and potentially many other parts of the network. The actors behind the intrusion used the technique to compromise FireEye and several federal government agencies, but it’s unclear how many other customers may have been affected, too. SolarWinds said fewer than 18,000 customers had downloaded the malicious update, but the company’s customers include some of the larger companies in the world, spanning technology, finance, banking, aviation, and many other industries.
The challenge for the security and IR teams at those companies now is not only determining whether the attackers accessed and exfiltrated any sensitive data, but also whether any of the systems connected to their SolarWinds deployment can be trusted. SolarWinds’ Orion platform is used to monitor a wide range of enterprise IT systems, and many organizations store credentials for those systems in the Orion database. So an attacker with access to that database would then have the keys to many of a target company’s internal resources.
“If you have that latest patch installed for SolarWinds, then you have to assume you were breached. You have to rotate those credentials as soon as possible. But the problem is it’s very hard to know what credentials you have stored in there,” said Rob Fuller, a security researcher who has worked extensively on SolarWinds for several years.
“SolarWinds is a beast. It’s huge. There are parts of it that don’t change. The RSA key is generated per customer, but it never changes after that. If I get access to the SolarWinds box just once, it’s essentially like a golden ticket. I can go back and talk to the database anytime I want and dump any of the data and any new credentials.”
Fuller released a tool on Tuesday called SolarFlare that’s been in development for several years and can be used to find and dump any credentials stored in SolarWinds Orion. SolarFlare was designed as a red team tool and Fuller said he’s used on many engagements in the past, including a recent one in which the organization had more than 200 sets of credentials stored in the Orion database. One of the tool’s capabilities is finding and reading the value of a cookie for the Erlang distributed programming system that’s stored in the Orion database. That value does not change over time and an attacker who was able to gain access to it would have system-level access to the other machines in the cluster. Fuller debated releasing the tool, but said he wanted IR teams and red teams to have the same capabilities to assess their exposure as the attackers in the intrusions seem to possess.
“That cookie would be the key to SolarWinds anytime you want it, as long as you can access the port it’s running on. If they were able to get that cookie out, they can get back into the box whenever they want,” he said.
“IR teams need to dig in on any use of the credentials stored in there and look for any kind of anomalies.”
"This is going to haunt us for a while."
But the challenges don’t stop with finding and rotating credentials, which is no mean feat in and of itself. The next issue is trying to determine whether the attackers accessed any of the other systems connected to SolarWinds, and if so, how to handle remediation.
The problem is you end being able to not trust any single component in the company. These are flat networks with phones and security cameras and door access everything else on the same network. Any one of those things could have to be fully redone, keeping in mind you probably want a lot of the data on those machines to move to new machines. It’s an enormous undertaking to do it right,” said Robert Hansen, CTO of BitDiscovery and a longtime security researcher who has helped companies recover from this kind of intrusion.
“It doesn’t take much for a determined adversary on a juicy target to pivot to the next target and the next one. It’s not just the data on those machines that’s suspect, it’s everything they had access to: API keys, GitHub repos, Salesforce. Anything you had access to from those machines.”
Treating that much of a corporate infrastructure as suspect makes daily operations difficult, and there’s the extra layer of the Sunburst backdoor used by the attackers perhaps lying in wait for months or years. Some of the C2 servers used by the malware have been taken offline, but that is likely not the end of the story.
“We’re lucky FireEye found this. A traditional company is not going to find this kind of thing for the most part. But what else is out there? What else can this malware do now that the C2 is offline? I’m not positive that every compromised box had human hands on it, but at the very least I’m sure a foreign adversary somewhere was cataloging what organizations were vulnerable. This is going to haunt us for a while,” Hansen said.