A lot of things happen on the Internet every day, and a great many of them are not great. Sometimes there are reasons behind those actions, and other times, well, things just go sideways for reasons that no one can really articulate.
That seems to be the case with the events surrounding the destruction this week of a small hosted email service called VFEmail. In the course of a few hours on Monday, an attacker compromised and destroyed the data on a wide range of servers operated by VFEmail, formatting the disks on productions servers and backups alike. The attacker, whose identity and motives remain unknown at this point, worked quickly and by the time the sun rose Monday morning, the operator of VFEmail was convinced the service was done.
“We have suffered catastrophic destruction at the hands of a hacker, last seen as aktv@94.155.49.9. This person has destroyed all data in the US, both primary and backup systems. We are working to recover what data we can,” a status message on the service’s site said as of 5 A.M. on Feb. 11.
How exactly the attacker was able to access the company’s servers, which were in several different data centers in geographically separate locations, isn’t clear. The operator of VFEmail said on Twitter that he was able to observe the attacker formatting one of the company’s backup servers, but by then it was too late.
“At this time, the attacker has formatted all the disks on every server. Every VM is lost. Every file server is lost, every backup server is lost. NL was 100% hosted with a vastly smaller dataset. NL backups by the provideer (sic) were intact, and service should be up there,” VFEmail posted on Twitter.
“Strangely, not all VMs shared the same authentication, but all were destroyed. This was more than a multi-password via ssh exploit, and there was no ransom. Just attack and destroy.”
For the last few days, the VFEmail staff has been working to restore the service, and some of the functions are back online. This kind of destructive attack, while rare, certainly isn’t unprecedented. Five years ago, code-hosting company Code Spaces was forced out of business after an attacker launched a crippling DDoS attack on the company and also compromised Code Spaces’ Amazon EC2 control panel. The attacker demanded a ransom, but with company and customer data deleted, Code Spaces’ owners chose to shut down instead.
In 2005 attackers compromised CardSystems, a credit-card processing company, and stole more than 40 million card numbers, which had been stored in plaintext. Soon after, both American Express and Visa stopped using CardSystems for processing and the company later was acquired. And the short, eventful history of cryptocurrencies and exchanges is replete with incidents that have resulted in bankruptcies, company failures, and indictments. The Mt. Gox disaster is the most well-known example, with that exchange imploding in 2014 after hundreds of millions of dollars in Bitcoin it held were lost.
“A lot of smaller companies could be taken down like this. These are fairly experienced people doing these attacks. All it takes is the wrong kind of attention from the right kind of people,” said Adrian Sanabria, vice president of strategy and product marketing at NopSec, and a longtime security practitioner.
“I’ve never seen a case where they think of a hundred percent of the things that can go wrong."
These attacks, including the VFEmail intrusion, illustrate not just how quickly things can go south, but also the importance of being ready. For anything. That is much easier said than done, especially for companies with limited financial and technical resources. And even for large enterprises with mature security programs, planning for intrusions, data loss, and other events can be difficult, time-consuming, and costly.
“Let’s be honest, not even all of the large companies are doing this right. It’s hard,” said Rich Mogull, CEO of Securosis, an analyst firm.
Sanabria, who has extensive experience in the financial services industry, said that even proper disaster recovery and incident response planning isn’t enough. Organizations need to put their plans into practice at some point to see how they work, what needs tweaking, and what needs to be scrapped. Without that practice, it’s impossible to know how, or whether, things will work. Sanabria cited one company that switches from its production systems to its off-site backup infrastructure for a week every quarter to ensure that it works as intended. It’s a rare practice, but one that could save a lot of money and effort when a real incident occurs.
“I’ve never seen a case where they think of a hundred percent of the things that can go wrong. Just thinking through disaster recovery isn’t enough. It’s only going to get you halfway there,” Sanabria said.
Even with a sizeable budget and ample personnel, it’s easy to make mistakes. Which is why more and more organizations are moving toward the cloud option. Securosis’s Mogull said that can be an effective way to manage risk, especially for companies without a lot of internal capabilities.
“I don’t think everybody gets this right, but it is getting better, because the more people rely on cloud services, the easier it is,” Mogull said. “All they need to worry about is securing their cloud accounts, and that gives you tremendous resiliency for very little cost.”