xz, tarred

As someone who has been messing with OpenSUSE Tumbleweed on top of his normal installation of Nobara Linux, I knew this was serious when I did an update yesterday, and it forced me to re-download nearly 3,000 packages in a single sitting.

(Tedium doesn’t do April Fool’s jokes. This actually happened.)

A rolling-release variant of Linux, Tumbleweed is a good distro, one that takes security very seriously while putting you up against the edge of what’s possible (and not feeling a little rough around the edges, like Arch can sometimes feel). But like many other distros, its makers found themselves freaking out over the weekend after it was realized that someone had dropped a backdoor into the latest version of xz Utils, an extremely common compression program. The backdoor added a payload into the SSH protocol that could have been broadly exploited. (For those with technical knowledge who want to see how it works, check it out here.)

For Windows or Mac users, this would be the equivalent of someone hacking how the ZIP format works. Many major software programs for the Linux ecosystem are distributed as xz-compressed tarball files. While also used on other operating systems such as Windows, MacOS, and FreeBSD, it is deeply integrated in the way Linux works. If broadly spread, this would cause chaos and essentially leave a backdoor in nearly all Linux clients, and by extension would have affected all the software that relies on Linux, from cloud apps to software-as-a-service platforms. By sheer chance it was uncovered by someone knowledgeable who noticed the SSH process running a little funny.

This is probably the biggest Linux-centric security problem we’ve seen since at least Heartbleed, a bug affecting OpenSSL which hit almost exactly a decade ago and was such a big deal that it got its own name and branding. As supply chain attacks go, it’s one of the most consequential, because it would have nailed basically every computer running a recent version of Linux—servers, desktops, and everything in-between.

And surprisingly, the system worked. The payload only lasted for a few days before it was caught and removed, and was only placed there because a malicious user named Jia Tan played an extra-long game to get it in, spending more than two years contributing to xz and seemingly using a sock puppet to convince the maintainer to bring on an additional help. That sock puppet, using the name “Jigar Kumar,” leveraged a public admission of mental health issues on the part of developer Lasse Collin to push him to bring in the additional assistance.

“It's also good to keep in mind that this is an unpaid hobby project,” Collin tried to emphasize in a thread that has been heavily scrutinized in retrospect.

Kumar wasn’t having it.

“The only progress since april has been small changes to test code,” the user replied in an unsympathetic tone. “You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo.”

(Alan Levine/Flickr)

Knowing what we know now, it is clear that Collin was being played, with his personal challenges and honesty being held against him—and that allowed an apparent state actor to burrow into the Linux project. It was only an extremely prominent Linux user—ironically, an employee of Microsoft—that caught it.

The scary part is, in the decade since Heartbleed, it perhaps feels like, on the surface, we have not learned very much. We’re still stuck with under-funded, under-supported projects that undergird the whole apparatus, with software like OpenSSL and xz not being front of mind despite the fact that they play incredibly important roles. The social engineering aspect of what happened to xz, which allowed a bad actor to maintain a role in the project for years, points to a lack of vetting in open source. Which, to be fair, strikes at the delicate balance FOSS plays. You want the projects to be open enough that anyone can take part, but because this technology is used by so many people, it has to be managed correctly.

Large companies can help, but they can only do so much. Each of those 3,000ish packages has a group of maintainers behind them, many working on the software during their free time, basing their work on donations in sometimes underappreciated circumstances. (On the plus side, the situation that affected xz has people talking about solutions.)

If the right infiltrator taps the right card, the whole thing crumbles.