It was the CVE-2023-4863 vulnerability that revealed how truly bad the problem of vendoring had become in the Nix repository, recalled Nix Packages maintainer Delroth (Pierre Bourdon) in his February 5 FOSDEM talk, “Remediating 1000s of untracked vulnerabilities in nixpkgs,” recently posted online.
A heap buffer overflow in the libwebp image decoder (v1.3.2) was found in the Google Chromium browser. Chromium sent out word for users to update immediately because there were exploits in the wild using this vuln.
And so the Nix folks swiftly patched their version of Chromium, and since the vulnerability was actually with libwebp, the Nix version of that library was patched as well.
But then, the maintainers started wondering, where else was that decoder being used? Nix is a tightly run package management system for the Nix-OS, a Linux distribution, allowing users to download over 80,000 software packages, utilities.
Certainly, additional packages, such as other web browsers, must have also bundled libwebp in their own open source applications, an inclusion known as “vendoring.”
Turns out there were dozens, or maybe even more. Without any tools to sort through the packages, it took the team about a month to track down only about half of what they estimated was out there. At least 10 contributors spent “hundreds of hours” updating packages.
Initially, they tried to replace every vulnerable instance of that library they found, but soon gave up, recognizing the enormity of the task. And while most the high-risk cases were updated, those packages less likely to be impacted have been delayed, maybe indefinitely.
The lesson is clear: when a vulnerability strikes, package and repository managers can no longer assume the problem is solved once they patch their canonical version of the problematic software.
And if the Nix founders figured this out about their own libraries, it’s pretty likely happening everywhere.
What Causes Vendoring?
Lazy programmers? Instead of declaring a dependency, and drawing on a system resource, they will just cut and paste the library right into their app’s own source directory. Sneaky-sneaky downstream a copy of an open source library in your own application library.
To be fair, there are some practical reasons for doing this: The chief benefit is that you never have to worry about an external dependency breaking your applications, because it goes missing or is updated in a way that somehow breaks your own app.
Keeping it in-house, after all, will keep things working forever and ever.
Until a vulnerability is found, that is. While system resources are readily patched when they are found unsecured, the copies of these resources buried in the directory of some app’s code will more than likely not be updated and thus remain open to exploitation.
How many of these clever programmers realized that by adding an external library to their own stack they have also taken on the responsibility of maintaining that resource, just as much as they are responsible for their own code?
The Great Divide
In the Nix community, at least, the situation with vendoring is “not great” Delroth admitted. And it’s a problem that can’t easily be fixed.
In wondering how bad the problem of duplicate libraries had spread, Delroth created a few tools to get a handle on the problem: the self-describing grep-nixos-cache and the more language-specific vendored vulnerability scanner.
Turns out there were 116 other copies of libwebp in various nixpkgs across the Nix repository. And it was far from the only image decoder being bundled: He found 237 copies of libpng and 253 copies of libjpeg scattered about. And the all-purpose zlib compression library was found in 761 different places.
And who knows how many of these, if any, are being updated. A few versions of libpng go back to 2004!
Go on! nix build it and get a binary from 2004! Does it still have vulnerabilities? You bet!
The Rust programming language has its own peculiar take on this problem, he noted, through its use of LockFiles. With LockFiles, an application will require a specific version of a resource, which is identified by a numeric hash. So while the program calls an external system resource, it requires a specific version of that may never get patched.
Delroth highlighted Rust crates packages with LockFile dependencies. Of the 1,844 Rust packages identified, 1,149 (62%) had locked-in dependencies, meaning in Rust-speak, that they couldn’t be changed.
“Upstreams don’t understand this, or don’t care,” Delroth said. And 40% of those packages (744) have high or critical security holes.
Of course, not all of these vulnerabilities are exploitable, depending on how they may be used, but some may be.
What Can Be Done about Vendoring?
Delroth admits the Nix community has not set down any sort of compliance or even guidelines to discourage programming from vendoring.
It is just considered good practice to use the external system libraries, but there is no law saying you have to. It’s open source after all.
Nor is there a even stated preference for building from source. Users can go to AppImages and download an already-compiled binary instead. “People are very creative about how to get binaries,” Delroth said, adding that Debian is the gold standard for keeping binaries fresh.
(And this is assuming someone upstream is still maintaining the packages. Many aren’t, Delroth admitted.)
For Nix, the best approach may be user education, says Delroth. and perhaps a new tag, something like knownVulnerable, that can be used to flag or even programmatically steer users away from out-of-date packages,
What Does Vendoring Mean for the Rest of Open Source?
Keep in mind, that the Nix folks run a pretty tight packaging system, comparatively speaking. So if vendoring is a problem here, you can bet it is a problem with other open source repositories. What about the other Linux or Docker distribution points, or the JavaScript jungle of frameworks, or even the Wild West frontier of C/C++ libraries?
In a 2022 survey by The Linux Foundation, only 18% of open source contributors reported that they are confident that the indirect (transitive) dependencies their organization depends on are not malicious or compromised (A transitive dependency is a dependency for a package that you use as a dependency).
Even when there is an existing policy in place for open source security, that figure only goes up to 27%.
“I think that people are concerned that if they report their usage of a dependency, then it will slow down their development and make them look bad,” TNS analyst Lawrence Hecht noted in a Slack conversation.
They have a bureaucracy to deal with as well. A Tidelift survey found that 56% of open source developers at enterprises with more than 10,000 employees say it takes more than a week to get approval to use an open source component.
Is this a problem that could be contained at the user level, through a software bills of material (SBOMs) that inventory all the one source components that could be used?
“It is highly dependent on how the code or software is vendored, and if the SBOM tool used to generate the SBOM can deal with that,” noted Michael Lieberman, co-founder of the security platform company Kusari and one of the creators of the GUAC SBOM visualization tool, in an e-mail.
The Go programming language, for instance, “handles vendoring in a fairly straightforward way,” he said. “As long as you don’t modify the vendored code, SBOM tools should be able to handle picking it up.”
Not every case is as straightforward as that, however. “People will sometimes manually vendor code by downloading the source directly into another project. As soon as you modify even a single line of a vendored project, you have now forked and modified that software to turn it into something else.”
In these cases, you run the risk of an SBOM tool being unable to identify the code. You broke it, you bought it.
The term “vendoring” seems to have been borrowed from the world of commercial software, where a commercial vendor would take control of a library for its own purposes. In the open source world, “every open source distribution point” is a vendor, noted Tim Mackey, the head of software supply chain risk strategy at Synopsys Software Integrity Group, in an e-mail.
Take the almost universally-used OpenSSL library, for instance. The official source is a GitHub repository, but most users get it from a Linux distribution such as Red Hat or Debian. The code may be exactly the same in both distros, but they were compiled for different targets (hence the name “vendoring.”). Realistically, there are probably thousands of different forks, or intermediate source code copies, of OpenSSL in the wild.
“This entire ecosystem makes patch management quite hard,” Mackey wrote. “A patch designed for one branch of OpenSSL might not work the same on a different branch because of changes I made to an independent fork of the code.”
A good SBOM will look at the package’s pUrl (persistent uniform resource locator), which identifies the province of a package.
“If that pUrl points to the correct GitHub repository for the version of OpenSSL used by the application, then we’re able to identify who is responsible for any patch. Similarly, if the version of OpenSSL originates from a distribution like Debian, then Debian is the responsible party for any patches,” Mackey noted.
An SBOM does not know anything about the code that was shipped in the resulting binary, however. To address this issue, VEX can be used to map the workflow of how an application was assembled.
“The SBOM should document the component and its version, but it’s up to the component vendor to determine if there is a vulnerability in their code and if it’s exploitable or mitigated without a patch,” Mackey wrote. “This is why VEX workflows are an important part of solving the vendoring problem.”
Delroth’s entire presentation can be viewed here:
TNS Analyst Lawrence Hecht contributed to this post.
The post Vendoring: Why You Still Have Overlooked Security Holes appeared first on The New Stack.
A cautionary FOSDEM talk from the Nix community about all the vulnerable software that may still overlooked on your systems.