When your home has been broken into, you may not initially comprehend all that has been taken, or the damage that has been done. This is the state of apprehension the Linux community now feels with the recently-unearthed xz backdoor security vulnerability.
“This upstream supply chain security attack is the kind of nightmare scenario that has gotten people describing it called hysterical for years,” Kubernetes Security Chairperson Ian Coldwater had written on X. “It’s real.”
A Microsoft engineer first detected the back door, which he traced back to a recent update to the xz compression library. The library update was a recent one, but it already found homes in the rolling and advanced “rapid” releases of some Linux distributions.
The back door takes a certain combination of conditions and dependencies to trigger. Once triggered, however, an attacker could enter your system without any authentication at all.
The errant code has been quickly expunged, but now questions linger as to the potential damage this backdoor has already caused — as well as who planted this subterfuge, and what their intentions were.
Even more concerning is the possibility of other heretofore undiscovered backdoors that have been planted in this library, or have taken root in earlier versions of the library that many more servers are still using.
If It Weren’t for One Nosey Engineer…
Thank God for engineers geeky enough to debug a slow log-in time on their SSH session.
Microsoft Principle Software Engineer Andres Freund had noticed his remote ssh log-in took 500ms longer than it should have. He traced the latency to a system call that SSH made, for some reason, to the liblzma compression library, which is included with the xz utility embedded in Freund’s Debian sid installation.
He tracked down the backdoor code in the xz utility tarball for that Debian used in the installation process — though they were not in the original GitHub source code for the library.
The extra baggage was an obsfucated script that would get executed at the end of the configuration setup for the tarballs.
Freund reported the foul tarball to Debian Security and then to the distributer’s channel. Red Hat submitted this issue as CVE-2024-3094, ranking it with a severity of 10.
That this injection of seemingly malicious code happened so far upstream in the Linux release cycle for concerned Freund.
“Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the ‘fixes’ mentioned above,” he wrote.
Who Is Jia Tan?
Red Hat engineer Richard WM Jones had been in contact with the apparent author of the backdoor, he relayed on Hacker News.
The contributor, who went by the name of Jia Tan, had been “trying to get xz 5.6.x added to Fedora 40 & 41” because of its “‘great new features’.”
From Jia Tan’s GitHub account.
“He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise,” Jones wrote.
It was found that the XZ Utils 5.6.0 and 5.6.1 release tarballs that contain the backdoor. Both were created and signed by Jia Tan (JiaT75).
Jia Tan is likely a pseudonym, noted security expert Michal Zalewski, explaining that the persona appeared apparently out of nowhere in 2021.
With no previous activity under this handle, JiaT75 signed up for GitHub in 2021 and went to work immediately on the projects on the xz utilities. The account has no identifying information beyond a Gmail address.
The xz chief maintainer has Lasse Collin (Larhzu), who has been with the project since its inception. He has typically signed the xz tarballs (a bundle of multiple files) for distribution. He let Tan handle these last few releases, however.
How much Collin knows about Tan is not clear. Just prior to this mess, Collin logged off the internet, in an online sabbatical, hopping on only once to post a short response on the project site.
Zaleski’s sleuthing had found that Collin in the past few years had been beleaguered by trolls hounding him to step down from his post as xz administrator.
Trolling the xz maintainer to step down.
In one message, Collin admitted that he had little time of late to keep up with the growing backlog of issues. “Something has to change in the long term,” he wrote, adding that he looked to Tan taking on more duties as time passed.
Zaleski suspects that JiaT75’s work, given its general high quality, is not one of a hobbyist.
“All signs point to this being a professional, for-pay operation,” perhaps even one by a foreign government, Zaleski surmised.
Other security experts seem to concur on the overall sophistication of the injected code:
“This might be the best-executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library,” wrote open source maintainer Filippo Valsorda on BlueSky.
Jia Tan only had access to the xz files hosted on GitHub; Collin retains control of the website. For safety, GitHub has since disabled all the xz utility repositories, and suspended the accounts of both Tan and Collin.
Is the xz Backdoor Planted on your Linux Server?
If you run Linux or the macOS systems you most likely have some version of the xz and the liblzma dependencies, which are needed to uncompress software packages for installation and updates.
Thus far, it is mostly rolling release and rapid update distributions that have ingested XZ Utils 5.6.0 and 5.6.1, such as Fedora Linux 40 and Fedora Rawhide and the Debian advanced distributions.
Ubuntu 24.04 LTS (Noble Numbat) also contained the infected files, which have since been removed. Advisories were also issued by Arch and openSUSE.
Red Hat has reported no versions of Red Hat Enterprise Linux have been compromised.
The backdoor appears to be triggered only through a select set of conditions: through a “remote unprivileged systems connecting to public SSH ports,” according to a za-utils backdoor FAQ posted by Gentoo Linux developer Sam James.
In addition to having either the 5.6.0 or 5.6.1 tarball installed, the exploit also has to be a Linux distribution running on AMD64 hardware, and uses the glibc library (such as all those Debian and Red Hat-derived versions).
A combination of systemd and a patched openssh also seems to be a requirement for the backdoor.
The payload gets triggered by a running version of the sshd daemon in /usr/sbin/. The malicious code actually gets embedded into sshd itself, thanks to a recent sshd patch to support systemd-notify allowing other services —including liblzma — to be alerted when sshd is running.
Once inside ssdh, the payload then redirects sshd’s decryption function to bypass user authentication.
“Other systems may be vulnerable at this time, but we don’t know,” James wrote.
Red Hat warned its users of the severity of the compromise:
“Under the right circumstances, this interference could potentially enable a malicious actor to break sshd authentication and gain unauthorized access to the entire system remotely.”
How Many xz Backdoors Could There Be?
Given these conditions listed above, if you are running a server instance of a publicly accessible SSH, James advised that you should “Update RIGHT NOW NOW NOW.”
He stressed that the knowledge about the backdoor triggers and versions they infect are extremely limited at this point.
“While not scaremongering, it is important to be clear that at this stage, we got lucky, and there may well be other effects of the infected liblzma,” James wrote.
For one, JiaT75 could have planted other, better-hidden, backdoors into earlier versions of xz during his tenure, which goes back to at least to v5.3.1?
This would mean, of course, that a much larger pool of Linux distros could be affected.
The U.S. Cybersecurity and Infrastructure Security Agency (CISA) is currently investigating the backdoor further, according to Red Hat.
The good news is that it could have been worse: Vanilla upstream OpenSSH isn’t affected — unless liblzma is added as a dependency.
Nonetheless, OpenSUSE recommends its Tumbleweed users reinstall SSH for public-facing servers, as there can be no telling if those servers have already been compromised.
At any rate, how the backdoor got so close to so many production systems may be a cautionary tale over the state of the internet infrastructure.
“I do however think that this should mean an end to the practice of preferring manually built upstream tarballs over pulling in git sources directly that distributions such as Debian have espoused,” one commenter noted on LXN.net.
“It’s the one weak link where few eyes exist in an otherwise pretty reproducible pipeline and it was really only a question of time until someone took advantage of it.”
The post Linux xz Backdoor Damage Could Be Greater Than Feared appeared first on The New Stack.
A mysterious contributor who planted the backdoor helped maintain the widely used xz compression library for the past two years. So what else was hidden in there?