With some software vulnerabilities, it’s not immediately clear how serious the issue is and what the long-term effects might be. The Log4j vulnerability did not fall into that category. Within a day or so of the initial disclosure, researchers and incident responders understood how critical the bug was and were beginning to grasp how long the tail of remediation and exploitation could be. Now, two months later, the end of that tail is still nowhere in sight, but the quick response by and collaboration within the security community mitigated the potentially disastrous effects of the bug tio a degree that may never be fully known.
The Log4j bug and the response to it share some similarities with previous large-scale vulnerabilities, such as Heartbleed, Spectre, Shellshock, and others, in that they all affected a wide range of vendors and applications and required broad response efforts from vendors, developers, government agencies, and researchers. While painful at the time, those responses and remediation efforts also served as training grounds for what would come after and enabled organizations to hone their processes and skills. In the case of Heartbeed, which emerged in 2014, it took Cisco 50 days to identify all of the applications and products that were vulnerable. For Log4j, the company identified vulnerable products and had patches available within 10 days. In fact, some fixes were available in less than 48 hours, Brad Arkin, chief security and trust officer at Cisco, said during a hearing of the Senate Homeland Security and Governmental Affairs Committee on the Log4j issue Tuesday.
“By focusing on historical lessons learned, building better and more secure software, and having data about which specific applications and software we use, we were able to move quicker, eliminate risk faster, and have the agility needed to manage our own security and resilience. For Cisco, the key differentiator was our improved visibility into the software applications and third party products that we use as a company. Additionally, we now use tooling to allow us to see the software maintenance status of the applications we use, identifying whether a particular piece of software is the latest version,” Arkin said.
“We’ve made a lot of investments to squeeze the timeframe down.”
"The information we got helped us to understand the techniques and attacks that were being used in the real world."
One of the major differences between the Log4j response and previous efforts was the existence and assistance of the Cybersecurity and Infrastructure Security Agency. The agency, which is the lead for federal cybersecurity efforts, was only established in late 2018, and coordination between the federal government and the private sector was not nearly as organized before CISA came onto the scene. In addition to directing internal government response efforts and prioritizing remediation targets, CISA also works with researchers, vendors, and affected organizations to coordinate industry wide sharing of information and threat intelligence. The recently established Joint Cyber Defense Collaborative (JCDC), which is a cooperative clearinghouse for information sharing between CISA, federal agencies, and private companies, now serves as the central nervous system of CISA’s response efforts. The Log4j incident was the first really large-scale effort since the creation of the JCDC, and participants said the involvement of the JCDC was a key factor in limiting the damage.
“The most recent JCDC engagement, which occurred after Log4Shell was first discovered, presents an important use case of the long-term opportunity this collaboration vehicle presents. It can be an exemplar of successful public-private sector cooperation - specifically, the JCDC working as a venue for commercial competitors to act as peers, and share rapidly developing situational awareness to help secure our National Critical Functions,” Jen Miller-Osborn, deputy director of threat intelligence at Palo Alto Networks’ Unit 42, said during the hearing.
The JCDC includes a number of large technology companies, such as Amazon, Cisco, Google, Microsoft, FireEye, and Palo Alto. Arkin agreed that the JCDC played an integral role in the Log4j response, but emphasized the need to have as much unclassified intelligence as possible in order to ensure that a wide range of organizations and individuals can access it.
“The information sharing from the JCDC definitely added value to our efforts. The key thing for us is, when there’s an infinite number of things you could be working on, how do you prioritize your efforts to where the bad guys are actually preparing to do bad things to your environment. The information we got helped us to understand the techniques and attacks that were being used in the real world,” Arkin said.
“The thing I think about that is most important when I think about threat information sharing with the government is keeping the classification level as low as possible. If I go into a briefing that’s at a very high level, if I can’t share that with the rest of my company, it’s not as useful as it could be.”