Worst Firmware Update Ever

Today in Tedium: As companies innovate over time, they inevitably take things away. In the case of Apple’s M1 chip, it took away the ability to upgrade basically anything. (I want one.) As I wrote earlier this year, the Apple Silicon shift is not its first move away from a CPU architecture, but its third, and the parallels between the transitions from PowerPC and Intel are interesting to watch in real time. Now, while it was the most influential company that offered PowerPC computers for sale, it was technically not the largest—not by a long shot. The winner was technically the Nintendo Wii, which sold more than 100 million units during its seven-year history on the console market, all while rocking a variant of the PowerPC G3 processor line used in many early Macs. As I wrote in my prior piece on Apple and PowerPC, the Wii’s primary competitors in home video games at the time—the Xbox 360 and the PlayStation 3—also used architecture based on the PowerPC. The PlayStation 3, while it did not best the Nintendo Wii, may have bested Apple at making the most popular PowerPC platform intended for a degree of general purpose computing … well, until Sony, under duress, threw that crown away. Today’s Tedium talks about OtherOS, or how Sony made the Linux and hacker communities mad. — Ernie @ Tedium

Before we get going, be sure to check out today’s sponsor, SetApp.

An example of Sony’s PlayStation 2 Linux kit, which uses the NeXT-inspired Window Maker interface. (Colin Keigher/Wikimedia Commons)

Sony’s surprising openness to Linux on the PlayStation … at first

It’s really fascinating to consider that Sony, the company whose competing interests in content and hardware have likely played the biggest role in the rise of digital rights management, that once literally installed a rootkit on people’s computers, was so willing to give Linux a first-class citizen status on its consoles.

It feels like it shouldn’t be in Sony’s DNA, right?

But Sony had tried to leverage its position as a video game player to help build goodwill among developers, and adding Linux support to the PlayStation 2 was a good way to do that. It also helped that Sony had developed a development kit for the original PlayStation called the Net Yaroze, with the goal of helping to spur interest in development among hobbyist communities. (This was in sharp contrast to Nintendo’s traditional stance, which often discouraged unlicensed development.)

It was a good idea—and Sony’s move to block off the DVD-ROM drive, while limiting the console’s capabilities as a Linux machine, seemed like a fair compromise.

But then, when Sony released its follow-up to the PlayStation 2, it did something unexpected: It included a way to natively boot into alternative operating systems. The PlayStation 3 included a functionality called OtherOS, which allowed for easy booting into other operating systems. Why was it possible to use a Live CD with the PlayStation 3 but an unacceptable risk with the PlayStation 2? It comes down to how the boot process was implemented. Simply, it used virtual machines, which were supposed to limit access to the full system. As you may know about virtualization, it naturally comes with a performance hit—but the belief was it would keep the PS3 secure while still allowing for non-gaming use cases.

Yellow Dog Linux, as shown running on a PS3. (via the Yellow Dog Linux forums)

With Playstation 3 devices capable of running PowerPC-based variants of Linux such as Yellow Dog Linux, it created opportunities to stretch the console into use cases that took advantage of its unique multi-processor Cell architecture, which was seen as potentially beneficial in supercomputing applications. After all, a PS3 might have been expensive for a video console upon launch, but it was cheap as the basis of a supercomputing cluster. Its main CPU unit was faster than the CPU in last-gen PowerBook, but also had a number of co-processors called Synergistic Processing Elements that, together, helped the architecture dwarf many personal computers at the time … in software that could take advantage of it.

It was a bet by Sony (and by extension, its chip supplier IBM) that it had created something so epic that it could draw in the technical community, along with potentially the next generation of developers. In a community Q&A with Slashdot, the president of Sony Computer Entertainment Worldwide Studios, Phil Harrison, said that the goal was to make room for developers to have a way to learn skills using real hardware, to encourage their improvement. He cited his own growth when doing the same thing with the Commodore 64, how it allowed him to try new things while still trying to learn how to program.

“Now, those industry doors are largely closed by the nature of the video game systems themselves being closed,” he said. “So, if we can make certain aspects of PS3 open to the independent game development community, we will do our industry a service by providing opportunities for the next generation of creative and technical talent.”

That’s all well and good, but there was still a rub: “Now having said all that, we still have to protect the investment and intellectual property rights of the industry so we will always seek the best ways to secure and protect our devices from piracy and unauthorized hacking that damages the business.”

And, unfortunately for everyone involved, that’s exactly what happened. And Sony did not look good by the end of it.

The “fat” edition of the PS3, which had the OtherOS option available. (Evan Amos/Wikimedia Commons)

Why Sony’s attempt to turn the PS3 into a computer turned into a case of corporate self-sabotage

Sony talked a big game upon the launch of the PS3, particularly when it came to whether the console was “hackable.” Sony very strongly implied it was not.

And for more than three years, that seemed to be the case. Well, until a hacker with a reputation decided to take a swing at it. The result killed Sony’s Linux efforts almost immediately and hurt the company’s homebrew-friendly reputation almost overnight.

The hacker that first found a way into the kingdom was Geohot, a.k.a. George Hotz. Hotz had gained a reputation as a master hacker after being one of the first to unlock and jailbreak an iPhone, and took it upon himself to find ways to exploit the console, which had a reputation for being tough to hack.

It took Hotz about five weeks to find a workaround, something he documented on a blog. The vulnerability, as you might guess by the fact that I’m writing this, involved the use of OtherOS. In a blog post revealing his success, he described how he attained hypervisor-level access to the system’s processor and full access to the system memory.

“Basically, I used hardware to open a small hole and then used software to make the hole the size of the system to get full read/write access,” he told The Register. “Right now, although the system is broken, I have great power. I can make they system do whatever I want.”

Geohot noted that in his initial announcement that there was a risk that Sony could remove features to rein in the hack.

“As far as the exploit goes, I’m not revealing it yet,” he wrote. “The theory isn’t really patchable, but they can make implementations much harder.”

He soon did release details on the exploit, and Sony responded with an aggressive firmware update. With Version 3.21 of the PS3 firmware, the company removed support for OtherOS entirely, citing security reasons, and informed users that if they did not agree to the upgrade, they would lose access to the PlayStation Network, games that relied on newer firmware, and access to copyright protected videos.

And because Sony apparently had no sense of irony, they literally released the update on April Fool’s Day in 2010.

(Side note: Poor Yellow Dog Linux, whose developers had banked its future on OtherOS being accessible … and were the first to reveal Sony’s plans to update the firmware.)

The loss of access to Linux on the PS3 didn’t affect most regular gamers, but the ones who were technically oriented were upset—and this slowly led to an escalation of energy against Sony in trying to break into the PS3.

Geohot played a role in this escalation—for example, helping to restore the OtherOS feature just days after Sony removed it. But he wasn’t alone. Others joined in, too.

Eventually, Sony brought in the legal team to take on Hotz, suing him and a secondary group, fail0verflow, for their efforts to jailbreak and reverse-engineer the console.

Hotz, at least at first, took the lawsuit in stride, posting a music video on YouTube where he rapped about the case, with a flow highlighting the fact that he is clearly a polymath.

But the legal action took some dark turns; at one point, a judge approved a subpoena that gave Sony access to the server logs and IP addresses of anyone who accessed Hotz’s blog, the one that detailed his exploit efforts. If it seems like it was over the top, that’s because it was.

Sony eventually settled with Hotz, who wrote this in a statement about the settlement: “It was never my intention to cause any users trouble or to make piracy easier. I’m happy to have the litigation behind me.” Some, such as the Electronic Frontier Foundation, took it to be something of a gag order.

(Hotz later took his hacking abilities into the startup space, launching the artificial intelligence startup comma.ai, which specializes in autonomous driving solutions.)

This decision to go after Geohot cost the company goodwill for years afterward.

As Sony had specifically marketed the first model of the PlayStation 3 as having this ability to boot into Linux, it meant that some tinkerers spent hundreds of dollars on the machine for a feature they could no longer use. Perhaps it wasn’t the PS3’s primary use, but it nonetheless gave it daily-driver capabilities for some. You literally could get work done on a PS3, which had hardware that was pretty good for its day, so good that some out there feel that the Cell architecture was never truly utilized in full. One day, all of it was gone.

This created a mess of legal action, as people sued over losing access to something that a subset of them really liked. It was like Sony had shown that it cared about the little guy, but as soon as the little guy did something it didn’t like, it pulled the rug out.

(One point of irony here: Not long after Sony sued Hotz claiming violations of the Computer Fraud and Abuse Act, the company faced a class-action suit accusing the company of violating the same law.)

Ultimately, the company agreed to a settlement, which took a few detours, but ultimately led to a class-action settlement for the roughly 10 million people that bought the original “fat” edition of the PS3.

That long-running class-action lawsuit could have been avoided had Sony taken another approach to handling OtherOS. It was as if all of the good things the company had done to embrace homebrew development had been forgotten overnight by a corporate culture arguably too focused on piracy, and people instead looked to the negative stuff like the heavy DRM and the rootkit scandal, and offered a heavy-handed dose of payback in the process.

Geohot’s formative efforts in jailbreaking the console eventually led to more in-depth hacking efforts by others that took the mantle, as Sony’s decisions made the PS3 a target. By 2012, a series of codes were released that allowed for the installation of custom firmware. To this day, you can mod an old PS3 to access (an admittedly older version of) Linux, and you can even do so with low-level hardware access OtherOS didn’t offer.

Sony’s efforts to stop its console from getting hacked led to it getting hacked more thoroughly than it might have … had it left Geohot and his blog alone.