Mike Hanley, CSO of GitHub, recently joined Dennis Fisher on the Decipher podcast to discuss the White House open source security summit in January, the Log4j response, and the challenges of helping millions of developers secure their projects. This is an edited and condensed transcript of the conversation.
Dennis Fisher: Was there anything that kind of surprised you after you got in there for a few months and were like, wow I wasn't expecting this or this is a little more difficult than I thought it might be?
Mike Hanley: There's a good preamble here which is sort of of like how did GitHub and I find each other and the timing was so fortuitous and I think it spoke to like a little bit of coloring the the experience of the last year for me. Back in December of 2020, you'll remember the time period, where we sort of went very quickly from hey, FireEye had a security incident, to everybody had a security incident, on that Sunday afternoon and evening when they had had that press call. Literally I think the next week was when I had my first conversation with Nat Friedman about hey what do you think about coming to GitHub to be to be CSO, and how I looked at that opportunity at the time, especially in the context of everything that was going on in the software space, was this is about to rocket into the mainstream in a way that we haven't seen before, specifically around software security and what a great time to be at the home for developers and open source software trying to make a dent from there. That feels like a good place to try to make a positive impact. As you know and can appreciate, that's the thing that matters to me most is people and impact. So it ended up being a really compelling set of timing but also an opportunity to come here and one of the things that Nat and I talked about early on was not so much that there were things that we needed to catch up on. But really I think we had a shared vision of the opportunity that could come from leaning into investing in security at GitHub and actually getting ahead. So for me the idea that we would triple the size of the organization to take advantage of opportunity was pretty attractive and compelling. So as a result It's been an intense year of hiring and developing the team and growing out some of the functions but totally delivered in terms of the types of impact that we've been able to have on not just protecting GitHub the business and GitHub the platform and making sure that that's a safe and secure place for people to come together and create and communicate with their communities and deliver and release the the projects and creative power of developers everywhere. But also to make sure that we can also help those developers deliver more secure projects and have the easy to use security tooling. That's so important to them to make sure that what they ship is something that has a certain set of security properties that it otherwise wouldn't have had without our help.
Dennis Fisher: You mentioned that one of the things that was attractive to you is that opportunity to have a big impact on a large number of people. It kind of makes me think of the early days of the Microsoft Windows push in the early 2000s. I'm sure that's probably one of the things that helps you attract talent too, is you can present an opportunity that not many other organizations can.
Mike Hanley: Yeah, definitely. I think both in the sense of GitHub is a unique platform to make a difference but also I think the team is unique in its structure and makeup as well, right? So CISO can mean a lot of different things at a lot of different companies in terms of how that's composed. I mean here I think it's a role that exists in sort of the most royal sense in that we also have things like a machine learning group that does anti-abuse at scale inside the GitHub.com group and that group is doing really interesting novel threat detection and prevention work on the platform and the kinds of challenges at that size and scope and complexity only exist in a handful of places. I mean, that function doesn't even exist in most security teams right? Because it just sort of doesn't have that category of problems. So that’s been helpful in terms of attracting, retaining and developing talent because you're just not going to see the types of issues that we've got to deal with but a few places that exist period on the internet.
Dennis Fisher: From the inside of GitHub, when the Log4j thing happened, what was it like?
Mike Hanley: Of course we like everyone else, once we heard what was going on immediately activated our incident response procedures. And one of the things that was probably unique for us is the team that curates the advisory database here at GitHub was involved from the beginning of the discussion because our priority was to make sure that we could protect our customers and developers who are counting on us and the security of the platform to help make sure that developers who were affected by the vulnerability got the information that we could provide them and that they needed from us to protect themselves and their own projects and their own customers and to try to help more broadly with coordination and sharing information with other folks who were trying to to get through this. I think it took a couple hours for us to realize that this was about as big of a deal as Heartbleed was in terms of breadth and potential severity and we were in a pretty fortunate situation because Alvaro, who's on our team, actually gave a talk on these JNDI issues back in 2016 at Black Hat. So we actually had one of the people who had sort of explored this bug class several years ago on staff and it was great because he was able to share information back with some of the maintainers who were working on the project and help.
So as we kind of worked through some of our incident response procedures looked probably a lot like a lot of other companies. It would sort of assess our exposure, figure out what we need to to fix you know, kick off the process of getting both the live production on GitHub.com squared away. But also our enterprise server customers squared away as well. One of the cool things was actually right before Log4j happened we had actually released a new GitHub code search platform and really novel leap ahead implementation or leap ahead improvements in terms of how people can search and discover code on GitHub.com and one of the early datasets that we used to index and develop that platform was GitHub's own internal codebase, and we'll probably write this up at some point here in the next few weeks and share the full detail of that, but this dramatically reduced the amount of time it took for us to fully assess where's our exposure, where do we use Log4jand sort of a prepackaged form versus something that might be a little bit more of a custom integration, and that was really helpful for making sure that we had a concrete understanding of where was it. What did we need to do about it from a first -party perspective? And getting that squared away so we were rolling mitigations that night out to the production live site.
Dennis Fisher: I know those discussions go on a lot in the security community in the developer community about how the open source maintainers and developers need resources. They could use money they could use code reviews or that sort of stuff. From your experience Mike, do the folks at the product level inside companies understand how dependent some of their products are on these open source projects?
Mike Hanley: Maybe. I mean to double back for a second on the point you were making, I think as an industry the more we can do to get support to those developers so that security just happens or happens in the context of what they're trying to do the better and I think for us. We see ourselves at GitHub in a particularly unique position to make an impact there and that's part of why we give away a lot of our security tooling for free to open source projects to help open source projects not ship bugs in the first place because they were using GitHub code scanning which we just give them. To the second part of your question about product teams and their understanding and appreciation of this, I think the good news is this is probably getting better in part because we're seeing more of the push on the regulatory aspects of things like SBOM. And I'm not saying SBOM’s a panacea. But it is one of those things that has an effect on people who need to sell software to generate revenue to run their business right? The public sector can create the market just by virtue of, for example, the federal government being the largest buyer of software in the world. But if they say they need SBOM for something then a business is going to have to provide those artifacts and figure out how to do that and then of course SBOM leads to at least some level of understanding about what goes into the thing that you are packaging up and shipping either as shrinkwrapped software or offering as a cloud service to government. So I think it's getting better.
"What will not make open source security better is solely top-down regulatory forces."
Dennis Fisher: Along the open source security lines I know you got to go to the White House for part of this open source security summit last month. First of all, what’s iit like when you get that email or a phone call? How many levels of, this is definitely a phish, did you go through before you thought it might be real?
Mike Hanley: Yeah that's a great question. A few. I think on the twenty-third my boss got a letter from the staff at the National Security Council saying, hey we we would like you to send your senior most security person to this open source security summit that we're going to have at the White House in the middle of January. So I got that note right before Christmas and then we spent the first week back in January basically working with the staffers at the National Security Council preparing to go to what was really a workshop. And you know I've got to give the White House and this administration a lot of credit for approaching it that way because sometimes you get these groups together and everybody's got 15 minutes to brief and there's discussion at the end and then everybody goes home and this was not that and I think that was an extremely wise choice to bring the participants together and then have it really be a working session where we're trying to tackle specific things get aligned on priorities and figure out how to leverage the public-private partnerships and also the key industry working groups that are focusing a lot of the private sector's efforts, like for example, the Open Source Security Foundation, so that we're all moving together. That was smart because what will not make open source security better is solely top-down regulatory forces and the great thing is this administration gets that and that's awesome and they're looking to figure out how we all leverage the unique strengths and positions that we're in to make an improvement on this? So GitHub's got some things that are uniquely in our control as the platform where most open source lives that we can do and then other organizations like Google or Microsoft or the Apache Software Foundation similarly have unique levers or capabilities or effects that they can create, whether it's financial or the ecosystems that they participate in and control or the open source projects that are under their curation. All of those things represent unique and important ways of complementing the things that the federal government is able to do through regulation through acquisition rules, etc. And so that was really encouraging. You've had these experiences too where you go into a room for one of these industry working groups and it feels a little bit like a family reunion where you know more than half the people in the room so that was great. So the conversation I felt like really hit the ground running on how we do this stuff together and partner with the Biden administration to really make meaningful forward progress on this.
Dennis Fisher: So what do you expect the concrete outcomes of this to be?
Mike Hanley: The White House invitation came at a time where we were already sort of kicking off the news cycle of the OpenSSF and myself and several other company representatives joined the OpenSSF board back in November and we actually had a board kick off out in Napa I believe it was in the middle of November so we had already been having discussions about what are the next big initiatives that we want to push together as an industry. So it was honestly like a continuation and there was a lot of good continuity between the conversations we were already having as a private sector with what the White House wanted to partner to accomplish on this, so that felt very natural. There's a lot of good work, like Microsoft and Google teams just announced with OpenSSF not too long ago the Alpha and Omega Project, which is really sort of two adjacent but related projects. One is to sort of go deep on this top list of open source projects that people depend on and help them tangibly and enduringly improve the security of their projects and then the Omega project is to sort of go broad and give everybody some of these substantial benefits and that's where again, there's already work for example, happening here at GitHub like giving away code scanning or secret scanning to open source projects that similarly has that sort of broad effect of rising the tide for everyone. So those types of things were already in motion and I think to be able to do those with the cognizance of the federal government on what we're trying to do and try to find ways to partner together with standards bodies like NIST or the National Security Council was sort of informing them on the perspectives that we've got those types of things I think just help make sure that we're continuing to all move this thing together because I think what's not helpful. You don't want people to just write a check and walk away or you don't want people to go to the Ivory Tower and say we're going to do X and then nothing happens after the meeting. I think the natural continuation of a lot of this work is already just sort of happening through the OpenSSF, which is really great and then some of the participants are still going to continue to do the things that are uniquely in their control.