Dan Lorenc, CEO and co-founder of Chainguard, joined Dennis Fisher on the Decipher podcast last week to discuss the rise of software supply chain security threats, the challenges of asset inventory and management, and the value of Sigstore for code signing. This is an edited and condensed transcript of their conversation.
Dennis Fisher: Where did the idea for what Chainguard is doing come from?
Dan Lorenc: I think the overall idea in the space of supply chain security kind of came up gradually. I was at Google for about nine years like you said I started there back in 2012 I think it was and worked on a bunch of different things throughout Google cloud platform. Kind of from backend infrastructure and then later out toward kind of open source developer tools in the container and Kubernetes space and Google in the 2012, 2013 timeframe that was right when big nation state attacks started happening to most of the big tech companies. I've heard similar things from Microsoft and Amazon. Pretty much all of the big tech companies started noticing. This was happening around then a lot of it's been talked about since at the time nobody was talking about it at all. It was top secret but it was kind of like a crazy revelation that you know in your job you might encounter nation States trying to attack you and you know they might even go as far as having folks go get jobs at the company and try to compromise it from inside and that kind of caught everybody in the industry by surprise back then. It's a little crazy to think about now when it happens all the time and it's pretty obvious, but back then it was new like folks weren't used to operating systems that way.
And so we spent a couple of years after that dealing with the fallout from realizing that you can't actually just blindly trust all employees to have access to all sensitive data when your company that size is dealing with that much sensitive information and systems had to be architected completely differently and baking in that culture of multi-party review and two-factor review to every single thing that happens not just access to production but code review, compilers kind of all of that stuff, and then when Kubernetes and containers and public cloud and Docker and everything started catching on a few years later and I started working on that, it was like stepping backwards like almost a decade and it was like wait a minute all that stuff we just built is gone now. Everybody's building stuff on Jenkins machines in closets and under their desks and nobody's tracking what goes into software and how it's getting built. And so that kind of made me pretty paranoid for some of the stuff I was building in open source and shipping and kind of led me down this rabbit hole. It was pretty boring for a while like yeah, nobody really cared about this at all and honestly just felt you're bothering everybody. Until SolarWinds happened honestly at the end of 2020, then it was like a night and day switch went off and everybody was like hey why haven't we been doing this for forever. It's so obvious in hindsight and so that's sort of how I got into this field, and the field kind of grew up.
Dennis Fisher: There was a lot of outward-facing changes that Google and other companies made, encrypting the links between their data centers and all that kind of stuff but it's cool to hear about the internal stuff too where you're looking inside and you're looking around and being like well why do we trust this system. Why do we trust this person?
Dan Lorenc: It's a big shift in the way you build systems and you know there's no perfect answer here. The best you can really do is have multiple people look at something in those situations because at the end of the day you are trusting people. Trusting people blindly is also terrifying especially when you're working on open source landscapes where you're taking code from anybody on the internet basically and if anybody has spent time on the internet, you realize that not everyone on the internet is a nice person and deserves your trust, and yeah it leads to kind of these inverted security setups in a lot of companies that we see too, just based on policies being applied. If you want to get a new vendor approved at a company you have to go through a crazy vendor approval process and security audits and budget approvals and all this stuff and it can take months. But if you just find an open source project on GitHub you can pull that in without asking anybody in most cases.
Dennis Fisher: You mentioned when the SolarWinds attack happened, which was the end of 2020, it seemed that was kind of a watershed moment for a lot of people in the security industry and also in the broader software industry I think too. They started looking at the dependencies and how many people had SolarWinds in their environment and how would they know if their version was compromised. So did you kind of look around and say, I told you. I was trying to tell you guys.
Dan Lorenc: Yeah, sort of. You know, a lot of it was like well you know you take this as an opportunity to do a tabletop exercise at your company. if this happened to us, like how hard would this be for us to detect and remediate and fix and do we actually have any controls in place that would have prevented this? A lot of organizations around the world are probably doing that around that same time and you know I've seen spreadsheets from CISOs of massive companies. They've showed me right after SolarWinds, that attack happened. You know we did an audit and you know we found 400 different Jenkins servers that were in use today across our company and it took us six months to do this and there's probably 100 more that have been spun up since then. And we really need to get a handle on this and it kind of raised that level of awareness to the executive level which is great, is kind of the only way you actually address something like this across the industry.
Dennis Fisher: Also I think there were a bunch of organizations that discovered that they had SolarWinds after that. I remember hearing stories from people that were like you know we found out four months later that we did have solar winds in our environment. We didn't even know.
Dan Lorenc: Yeah. Accurate asset inventory, accurate asset management, shadow infrastructure, kind of all of those things are a prerequisite to even being able to get started on supply chain security and a lot of folks are still struggling there.
"If anybody has spent time on the internet, you realize that not everyone on the internet is a nice person and deserves your trust."
Dennis Fisher: If you don't know what you have there's no way you can secure it. Are you still having to sort of stress the importance of that to potential customers when you're speaking to them?
Dan Lorenc: Exactly. Yeah I think if you followed the supply chain security space at all, you probably have heard the term SBOM, software bill of materials. You know it's being touted as one of the ways to help and improve supply chain security. And for anybody that doesn't know what it is and software bill of materials is mostly what it sounds like when you buy a piece of software. The vendor will include a bill of materials just like when you buy physical goods explaining what components and subcomponents and libraries are in there. The first part though is if you don't know what package software you bought in the first place is still running and where it's running then all the SBOMs in the world won't help. They're kind of one level removed from that in the SolarWinds case. Yeah, if you didn't know SolarWinds itself was running then it wouldn't matter that you could look up the SBOM or something and so getting an accurate asset inventory really is step zero. And any of these programs, if you don't know what software is running, you can't do patch management. You can't do vulnerability management. You can't ensure that it was built recently, you can't ensure that it was built securely. It's not easy in a lot of organizations to do that. Shadow Infrastructure is real. You know every customer we get into when we start putting in some of these asset inventory tools, they do find surprises. Everybody always finds surprising things in production. They don't know how that got there. It's scary in a lot of ways. SolarWinds obviously was the first big attack but one year later and we're coming up on the one-year anniversary of that again. So two years from the original one was Log4j, the Log4Shell vulnerability. it's about a worst case scenario vulnerability in an incredibly widely used component. That was about as easy to exploit with the most severe exploits possible. You know, kind of a worst case scenario from a vulnerability perspective. It's on the list of things attackers are going to try every single time and it doesn't have to work every time.
Dennis Fisher: When a lot of people think about software supply chain security, they're thinking about something like SolarWinds or Kaseya, where there was an attack that compromised a build or created a malicious version or something and that got pushed downstream. But Log4Shell is I think the much more insidious and problematic version.
Dan Lorenc: They're both big challenges right? And I think it's part of the confusion in the overall supply chain security space, is all these things get lumped together in one term. But they're completely different right? You know the attack on SolarWinds and Log4J are completely different things. You can't even call Log4J itself an attack. It was just an unintended bug that got found. So very different root causes. Very different solutions. Very different threat models too and folks are kind of struggling to wrap their head around the differences there and I think yeah to your point folks either jump on one side or the other side. Heartbleed was kind of the first eye-opener for most folks that hey, open source is everywhere and all code has bugs and some of those bugs can lead to security consequences. We should try to fix those and it's great that folks are trying. We're not going to fix them all, right? It's just software, right? You can't fix all the bugs. We should try though.
Dennis Fisher: I wanted to ask you a little bit about the Sigstore project, because I know you guys are sort of involved in it and I know Google is a big supporter of it. So what's the simplest way to explain to folks how Sigstore works?
Dan Lorenc: Sure I can start out with a high-level explanation and then talk about some of the history of it.. So if you're familiar with Let's Encrypt, it is a free service that gives out TLS certificates for websites. if you remember the internet before maybe five or six years ago most websites still did not have those certificates. You'd get little red x's when loading those or , it would be http only. And that was really dangerous right? If you logged into your bank without this then like you're sending your password over plain text on the internet and anybody in the middle can read that and do bad things. These certificates weren't new. They've been around since like the early 90 s but they were hard to get because it was manual. They fix this by building a new standard for websites to automatically go do all of that kind of exchange so nothing manual over email anymore. You could prove that you were the owner of a website. Automatically, they'd give you a certificate. Because it was all automated. They were only good for a couple weeks instead of years. So if somebody stole one. The damage was more limited. Adoption of TLS went from 50% to like 95% or 99% over just a couple years when it was made easy for folks. So we've been talking about Let’s Encrypt. How this relates to Sigstore, with Sigstore we try to do that very same thing but for signing code. There's also existing code signing certificates. You're trying to prove which person or company produces some code rather than encryption. But for the most part, the sort of certificates look the same. You can go buy one today you can pay a couple hundred bucks, send them a government ID, they'll send it back to you. It's good for a few years but for the most part, no one in open source was doing that and nobody is doing that outside of the kind of walled gardens where it's required. You need to do that to publish Windows drivers or you need to do something similar to publish an app to the iPhone store and this technology is around. It's just hard to use so we thought, how about we try to automate that, make it easy, make it free for developers and see if folks want to do it and so that's where Sigstore came in. Sigstore runs a free certificate authority. There's some cool technology with stuff like transparency logs and Merkle trees to make it all work and trustable. But at the end of the day a developer can run a command, their browser pops open. They verify their identity with, you know, an email address or a GitHub account or any common identity provider. They get a certificate tied back to that and then they're good to go. They can sign their code with it so you can use this with container images and npm packages, python packages, Java packages, pretty much any open source artifact distribution mechanism now supports Sigstore.