A critical vulnerability in the SaltStack Salt server monitoring and configuration management tool that allows remote code execution has drawn the attention of attackers over the weekend, resulting in at least three significant intrusions.
The vulnerability (CVE-2020-11651) was disclosed publicly on April 30 by researchers at F-Secure Labs and SaltStack had released updated versions that fixed it the previous day. However, some enterprises that deploy SaltStack’s version of Salt and didn’t update their implementations right away found themselves targeted by attackers who exploit the vulnerability to install a coin-mining script. Both Ghost, a content management platform, and Xen Orchestra, an orchestration platform for Xen servers, were hit by the attacks on Sunday. In both cases, some of the companies’ services were unavailable for a period of time, but neither one reported that any sensitive data or customer information was accessed or affected. There is a second vulnerability in SaltStack Salt, discovered at the same time by F-Secure, that allows directory traversal by an authenticated user.
“At 1:18AM, Sunday, May 3: some services on our infrastructure were unreachable. Only a subset of them, but almost at the same time on various virtual machines in various datacenters. High CPU usage was also another visible symptom,” a timeline of the attack from Xen Orchestra says.
“At 11:30AM, the dev machine was displaying the same previous symptom again. But now, it was visible live. The culprit was identified quickly as a ‘rogue’ Salt Minion process mining coins (hence the CPU usage, a process that also stopped web servers). It didn't happen again on other VMs because Salt Minion was disabled on them. This is when we understood SaltStack CVE-2020-11651 and CVE-2020-11652 were the attack vector.”
Salt is an open-source project created by SaltStack that enables infrastructure management in complex environments and is used widely in data centers and cloud environments. Server admins can use the framework to monitor server status and send updates, as well. The Salt framework uses a minion and master architecture, wherein the master gathers status messages from the minions and sends them commands in return. The remote code execution vulnerability in Salt versions 3000 and earlier and 2019.2.0 and earlier lies in the way that the master process handles some requests.
“The ClearFuncs class processes unauthenticated requests and unintentionally exposes the _send_pub() method, which can be used to queue messages directly on the master publish server. Such messages can be used to trigger minions to run arbitrary commands as root,” the F-Secure advisory says.
“The ClearFuncs class also exposes the method _prep_auth_info(), which returns the ‘root key’ used to authenticate commands from the local root user on the master server. This ‘root key’ can then be used to remotely call administrative commands on the master server. This unintentional exposure provides a remote un-authenticated attacker with root-equivalent access to the salt master.”
"We trusted its communication protocol (using keys) and we used that flexibility to add new VMs dynamically, without using a VPN or a secure tunnel"
The attacks that hit Ghost and Xen Orchestra were relatively simplistic and appear to have only installed cryptocoin mining scripts on the exploited machines. The exploitation attempts look to be coming from a coin mining botnet and there are several exploits for the code execution flaw available already. In its account of the attack, Ghost described a scenario that was quite similar to the one at Xen Orchestra.
“Our investigation indicates that a critical vulnerability in our server management infrastructure (Saltstack, CVE-2020-11651 CVE-2020-11652) was used in an attempt to mine cryptocurrency on our servers. The mining attempt spiked CPUs and quickly overloaded most of our systems, which alerted us to the issue immediately,” the Ghost timeline says.
“At this time there is no evidence of any attempts to access any of our systems or data. Nevertheless, all sessions, passwords and keys are being cycled and all servers are being re-provisioned.”
SaltStack released fixed versions of the affected software, versions 3000.2 and 2019.2.4. F-Secure researchers said they had found more than 6,000 instances of the vulnerable service exposed to the Internet.
“SaltStack is used to push updates or configure multiple VMs at once. We trusted its communication protocol (using keys) and we used that flexibility to add new VMs dynamically, without using a VPN or a secure tunnel. This was a mistake: it allowed the attackers to access the available Saltmaster port to inject their payload in available minions,” Xen Orchestra said in its description.
LineageOS, a replacement OS for Android, also was hit by an exploit for the SaltStack flaw.
"Around 8PM PST on May 2nd, 2020 an attacker used a CVE in our saltstack master to gain access to our infrastructure," the company said on Twitter.
SaltStack executives said the number of vulnerable systems exposed to the Internet discovered by F-Secure represent a small fraction of the Salt install base, but encouraged every organization running the vulnerable versions to update.
"Although there was no initial evidence that the CVE had been exploited, we have confirmed that some vulnerable, unpatched systems have been accessed by unauthorized users since the release of the patches," said Alex Peay, senior vice president at SaltStack, said in a statement.
We must reinforce how critical it is that all Salt users patch their systems and follow the guidance we have provided outlining steps for remediation and best practices for Salt environment security. It is equally important to upgrade to latest versions of the platform and register with support for future awareness of any possible issues and remediations.