The discovery of a backdoor in some versions of the XZ Utils open source tool last week averted what could have been a widespread compromise in the Linux ecosystem and prompted intense analysis and discussion in the security and open-source communities. While much is known about how the long-term operation resulted in the malicious code landing in the package, plenty is still in doubt. Decipher editors Dennis Fisher and Lindsey O'Donnell-Welch discuss three of the biggest open questions tied to this situation.
Dennis Fisher: There's a lot that's known about this XZ backdoor but we wanted to talk about a couple of things that maybe we still don't know about what happened and I think there's probably a bunch of them but let’s try and hit two or three of the bigger ones. I think probably the biggest one at least for me is, who is Jia Tan, this persona that was responsible for adding the back door to the XZ Utils a few weeks ago. I've seen a lot of speculation about it, but I don't think anybody has really come up with a concrete answer to this. It may be more than one person using that persona. It could be an entire team. Nobody really knows because there's other sock puppet accounts that sort of popped up to amplify that persona's message in GitHub and other places too. I've seen a lot of speculation about whether this was Chinese operators based on time zones, people think that maybe whoever this was changed their time zone on their commits to central European to sort of throw things off. I really haven't seen anything concrete that's laying out evidence that shows us exactly who this persona was.
Lindsey: It's really interesting. That's the big question of the week and it's been really eye opening to see all the activities that have been unearthed by the GitHub sleuths of what this account has done since 2021 I think was when the account first appeared and then, over the years just really took the time and effort to build up that trust in the open source community. And it's really this person or people who really legitimately appeared on the outside as a trusted member who was interacting with other people in the community. I think I saw a stat that the account had made like six thousand code changes or something over the years so it's not like a one-off account like some of the others. This was someone who really embedded themselves into the community and tried to build up that reputation before they ultimately did become a co-maintainer for XZ and then kind of moved forward from there.
Dennis Fisher: I think for me the most plausible explanation is this is an intelligence agency somewhere that really plotted this out and took their time as you just described. They probably had several people you know on this team working on this issue over the years. That persona landed several sort of innocuous or actually helpful patches to XZ over the years which helps build that trust relationship you just mentioned. That's the type of thing that you would think that an intel agency would do. They have time, they have technical resources, they can plot this out over a long timeline and see if it bears fruit. If it doesn't, what have they lost? Nothing really, aside from people's time. And that doesn't bother them.
Lindsey: And what sticks out to me too is this account also was working with some other projects outside even of XZ. So that's not great. They were looking at what they could get away with or just, as I said before, trying to build up that trust in the broader community. So there's a lot there.
Dennis Fisher: That sort of brings up another one of the things I don't think we really know, which is whether this has happened before. When I was talking to Dan Lorenc on the podcast this week and all the really smart technical analyses I've seen, the answer is probably yes and we just don't know about it. This same team may have had several parallel projects going on that could still be going on. Why put all your eggs in one basket? Why not try this across ten or twelve different projects? If one of them bears fruit, great. If several do, even better. We got super lucky that somebody found this before it actually got loaded into any Linux distributions. And sometimes it's better to be lucky than good.
Lindsey: Yeah I mean it's like what Dan Lorenc said in the podcast that you did with him earlier this week, he said it was inevitable, and I agree with that. It's such a ripe environment for threat actors to close in on and swoop in and gain that trust. It's kind of a terrifying thought. Obviously we found that one, which was great, but this is either happening right now or has happened and you just don't know, because when you look at the backdoor at the more technical level, they were very thoughtful about not just what they were putting in but how they introduced it.
Dennis Fisher: Yeah, the timeline is the craziest thing to me. Three years, and it could have been even longer. We just don't know, we're not 100 percent sure. But even that's an insane amount of time to spend on something like this, but the payoff could have been really great for the attackers. It could have been a complete catastrophe for Linux users. But, if you're an APT and you're not trying to do this stuff, you're bad at your job. This is what your job is.
Lindsey: Even how the backdoor was introduced, different pieces of it were introduced over the time span of multiple commits. How do you even approach the different puzzle pieces?
Dennis Fisher: It was extremely subtle. It took a very smart person looking at a very weird anomaly that had nothing to do with the backdoor itself. It was just a case of, why is there a lag on my machine on this one utility and he somehow went down a rabbit hole and found it. So it's kind of amazing. So I think the third thing that's probably the biggest question in my mind is what this means for the broader open source maintainer community and open source security as a whole. It's extremely silly to point at this maintainer and say this is his fault, that he could have done X, Y and Z to prevent this. I haven’t really seen any smart commentary saying what could have been done. This is a really well-planned, well thought-out operation and one person maintaining this project. He's not expecting this. He's not defending against intel agencies on a regular basis. If this had happened in a closed source or proprietary package, probably no one would have found it or if they did it would have been far, far down the line.
Lindsey: Yeah I think that's a really good point. I've seen a lot of other people point to the fact that it was in open source software was actually helpful. I was thinking I've seen a lot of parallels between this and Log4j and obviously they're both in this same kind of community, but it's incredibly different in a way because with Log4j that was more of a vulnerability and this is more of a very targeted attack. It's really difficult to know how this could have been prevented and what this means for the OSS space in general. But It's a big question mark still for me.
Dennis Fisher: I think if an intel agency decides that they're going to target your software project or your organization, they're going to win. They have time, they have money, they have resources. You're at a massive disadvantage, no matter who you are and what your resource level is.