The malicious code found embedded in versions 5.6.0 and 5.6.1 of XZ Utils last week appears to be the product of a carefully crafted supply chain attack that took several years to set up, security researchers said.
The malicious code (CVE-2024-3094) drew attention because it could allow remote, malicious actors to break sshd authentication and gain unauthorized access to impacted systems. The malicious versions of XZ were released in February, and Microsoft software engineer Andres Freund accidentally discovered and posted about them on Friday.
As security researchers begin to unravel the mystery of how the malicious code was introduced in a popular set of data compression software tools used in most Linux distributions, the name tied to a maintainer account for XZ that issued the commits for these malicious versions - Jia Tan - is at the center of many questions. A closer look at Jia Tan’s GitHub history reveals a years-long history with XZ, suggesting that the person or people that introduced the backdoor took multiple years to create a credible reputation and lay the social engineering groundwork for the attack.
“This wasn’t [a case] where there was a violation of the integrity of the supply chain where this was inserted at some point before it got into the final application,” said Feross Aboukhadijeh, founder and CEO of Socket. “This was from the source, the person who was legitimately authorized to publish this was the one that did the attack, which makes it super challenging.”
Playing the Long Game
The username under JiaT75 was first created on GitHub account in 2021, according to a detailed account of the timeline developed by security researcher Evan Boehs. Jia Tan then began to contribute to related projects, but it wasn’t until five months later, in April 2022, that the account submitted a patch for XZ via a mailing list, according to Boehs.
At that point, several other accounts - under personas by the names of Jigar Kumar and Dennis Ens - began to complain about the delay in the patch being merged and pressuring the project’s original author, Lasse Collin, to add another maintainer. Researchers pointed to this interaction as a critical part of the social engineering behind the campaign, as the original author likely felt pressured to add another contributor.
“The exploit within the xz compression tool involved a series of manipulative contributions and patches by new, suspicious accounts over several years, leading to a backdoor that compromised the tool’s security,” said Michael Skelton, VP Security Operations and Hacker Success for Bugcrowd in a post this weekend. “Through strategic pressure on project maintainers and subtle code changes, these contributors gained trust and inserted malicious code.”
Several days later, Jia Tan began as a regular contributor to XZ. While there are no further public details of how and when Jia Tan became a trusted maintainer for the repository, the account’s first commit was Jan. 7, 2023, and after that the primary email contact in Google’s oss-fuzz was updated to Jia Tan’s rather than that of Lasse Collin, according to Boehs. Over the course of 2023, Jia Tan continued to build trust while setting the groundwork for the backdoor. In March 2023, the testing infrastructure used for the exploit was committed by Jia Tan (despite Lasse Collin being attributed as the author), and in February 2024 the commits for the versions with the malicious code were issued, according to Boehs’ investigation.
Open Source Software Challenges
Skelton said that the incident “highlights vulnerabilities in open-source project management and the critical importance of thorough code review and maintainer support.” The incident was found by Freund not through any sort of official review, but by accident. Freund noticed that sshd processes in the liblzma part of the XZ package were using “a surprising amount of CPU” while he was benchmarking postgres changes.
Because Freund discovered the issue somewhat quickly after the malicious XZ versions were pushed out in February, only certain Linux distributions were affected - including Fedora Rawhide, Kali Linux and OpenSUSE’s Tumbleweed and MicroOS - and none of these distributions are stable.
Still, the incident sheds light on the inherently challenging nature of trust in the open source environment, which is run by developers and maintainers with limited time, funding and resources, especially as it relates to security. The difficulty of securing this ecosystem has previously been highlighted by different flaws in open source software over the years, including Log4j in 2021. Plans to tackle open source software security issues have been proposed over the years - including CISA’s proposal last year for a framework to help prioritize security risks in open source software that it will use to assess - and where necessary, publish alerts for - threats to critical open source software dependencies. However, a well-planned attack such as the one against XZ is difficult to defend against.
“Our processes for how we vet dependency updates is not very good as a community,” said Aboukhadijeh. “If you just look at it, if someone was paying close attention, you would see it. But we don’t have very good practices in place to vet this.”
To help organizations and maintainers determine whether a given package is affected by the XZ backdoor, security firm Binarly has created a free tool called xz.fail that will check binaries for the malicious code.