Splitting Machines

Why nobody thought x86 could do something as technically impressive as virtualization

In the 1970s, during the era of the mainframe, a concept came to light that seemed to speak to a future where software and hardware lived hand in hand: The hypervisor. A term coined by IBM that also stands for “virtual machine monitor,” it represented the idea of splitting up a computer into multiple parts that were theoretically separated from one another.

(Why “hypervisor”? Easy. The concept is essentially a reference to the supervisor above the supervisor. What, did you think it was something more futuristic than that?)

What might that look like? A 1971 article by IBM employee Gary Allred lays it out:

The Hypervisor concept was relatively simple. It consisted of an addendum to the emulator program and a hardware modification on a Model 65 having a compatibility feature. The hardware modification divided the Model 65 into two partitions, each addressable from 0-n. The program addendum, having overlaid the system Program Status Words (PSW) with its own, became the interrupt handler for the entire system. After determining which partition had initiated the event causing the interrupt, control was transferred accordingly. The Hypervisor required dedicated I/O devices for each partition and, because of this, the I/O configurations were usually quite large, and, therefore, prohibitive to the majority of users.

So, unlike a modern virtual machine, it was effectively running two machines completely separated from one another, as if they weren’t connected.

During the mainframe era, a use case like this made sense, especially given that a System/360 Model 65 was about the size of a vending machine.

But as we all know, computers kept getting smaller and smaller from there. First, bigger than a bread box, then smaller than a bread box, then about the size of a loaf of bread, and now about the size of bread. (Basically the bread box metaphor has been utterly destroyed by the smartphone.)

For a time, smaller computers meant smaller computing capabilities, no matter how fast the processors got. But that wasn’t necessarily the only factor at play.

Going back to our knowledge of RISC and CISC processors, we know that x86 processors made by Intel and AMD tended to have a lot of instructions, which made processors overly complex. This turned out to be a problem when it came to Gerald Popek and Robert Goldberg’s set of theorems. Intel had designed the x86 chipset to run software with different levels of privilege, set up in a series of “rings,” with the goal of limiting the attack surface of the kernel.

This is good for having a secure system, but less good if your goal is to run a copy of Linux inside a copy of Windows 2000. And because the structure of this system limited access to the number of commands software could have access to, it meant you couldn’t virtualize it in quite the same way as IBM virtualized the System/360.

Sure, it was technically possible to emulate a Linux machine on a Windows 2000 machine, but that introduced a lot of overhead. You were recreating hardware in software—likely the very same hardware that the system already had. This didn’t make any dang sense.

Intel wasn’t alone on this front—it was really a problem with every major platform of the time, casting a broad net—but given how broadly used it was, Intel was the poster child. The x86 processor set violated the Popek-Goldberg theorems in more than a dozen distinct ways, which seemed like it might just make VMs infeasible with the infrastructure the world currently worked on.

There were some emerging suggestions it didn’t have to be this way, though. On the Mac side of things, Apple had pulled off something of a magic trick with its transition from 68000 to PowerPC by making a translation layer that more or less ran old code natively. At first, Apple used emulation, but later switched over to something called dynamic recompilation. (This technique was developed by Apple employee Eric Traut, who later worked on the famed PlayStation emulator Connectix Virtual Game Station.) Put another way, the system was reprogramming the vintage code for MacOS to run in real time. It was so good that even in later PowerPC versions of the classic MacOS, much of the software was emulated 68k code. They didn’t need to update it, because it just worked.

Meanwhile, the Java programming language helped to popularize the concept of just-in-time compilation, which made it possible for a program to execute on a machine in real time without necessarily being tethered to its architecture. (Much of the discussion around VMs in computing magazines in the mid-’90s was focused on Java, partly for this reason.)

JIT is arguably one of the most important techniques in modern programming—your web browser uses it heavily to speed up its use of JavasScript—and a key element of what makes modern virtualization tick.

JIT proved that it was possible to translate applications in real time. But translating entire operating systems? The x86 was too broken to allow for anything like that.

The result was that the hypervisor, a somewhat forgotten remnant of the mainframe that had yet to overcome the Popek-Goldberg theorems that sidelined modern computers, had yet to make its big return to the mainstream. But a computer science experiment run on a top-of-the-line SGI Origin 2000 server was about to change all that.

Disco ball: The SGI-driven experiments that gave us VMware

If Backrub—the foundation of Google—was the most important research experiment happening at Stanford University in the mid-1990s, Disco—the foundation of VMware—was very much the second-most-important.

Built by PhD students Edouard Bugnion, Scott Devine, and faculty advisor Mendel Rosenblum, Disco was an attempt to solve the challenges that prevented virtual machines from living up to their full potential. In Disco: Running Commodity Operating Systems on Scalable Multiprocessors, their research paper on this work, the team spoke of sharing resources between with the host computer. From the 1997 paper:

Disco contains many features that reduce or eliminate the problems associated with traditional virtual machine monitors. Specifically, it minimizes the overhead of virtual machines and enhances the resource sharing between virtual machines running on the same system. Disco allows the operating systems running on different virtual machines to be coupled using standard distributed systems protocols such as NFS and TCP/IP. It also allows for efficient sharing of memory and disk resources between virtual machines. The sharing support allows Disco to maintain a global buffer cache transparently shared by all the virtual machines, even when the virtual machines communicate through standard distributed protocols.

That’s a big shift from requiring dedicated hardware for each separate machine as used in the System/360 days. But computing had the benefit of experiences like remote desktops and emulators like SoftWindows and VirtualPC to show that we could virtualize the context and reuse the components.

A little more about the Disco experiment: It’s rooted in Rosenblum’s work on something called SimOS, a prior initiative which Rosenblum and other researchers built on IRIX to experiment with completely simulating a computing environment through software alone. It was a project designed to help the university design a processor of its own, Flash, but it proved key to building a virtual machine and a thin OS layer, called SlimOS, to manage everything.

(Why Silicon Graphics? At the time, multi-core computer processors were fairly rare, and the SGI Origin 2000 was one of the few options on the market that could handle such a task. It likely gave them enough additional headroom to do more extreme testing.)

And the testing proved the model. The overhead was fairly modest given the performance, which meant that it would theoretically be possible to use hypervisors to split up a machine into multiple pieces, each of which controlled a different process.

And, famous last words, the paper ends like this:

This return to virtual machine monitors is driven by a current trend in computer systems. While operating systems and application programs continue to grow in size and complexity, the ma-chine-level interface has remained fairly simple. Software written to operate at this level remains simple, yet provides the necessary compatibility to leverage the large existing body of operating systems and application programs. We are interested in further exploring the use of virtual machine monitors as a way of dealing with the increasing complexity of modem computer systems.

The paper created waves. Within a year of its creation, Bugnion, Devine, and Rosenblum would become three cofounders of VMware. Rosenblum’s wife Diane Greene, an engineer who at one point worked at SGI, would become the fourth.

Just two years after the paper’s release, VMware hit the ground running: After a year in stealth mode, VMware Workstation, a Disco adaptation for Windows and Linux, made a huge splash at the start of 1999. (One early InfoWorld article featured a quote from a skeptical IT advisor. Little did they know that it would come to dominate their lives.)

Decades later, the researchers created a follow-up paper explaining their process for developing the original version of VMware workstation. The paper, which came years after the founders had left the company, explained that the x86’s convoluted instruction set and the diverse peripheral ecosystem created significant added complexities. (Yes, they knew that the theorems would make what they were trying to do difficult.)

But so too, did the operating systems, which they had to implement one at a time. Linux was easy. Windows was hard. OS/2 was impossible.

As the authors wrote, IBM’s failed attempt to compete with Microsoft was just too proprietary to be feasible:

Although our VMM did not depend on any internal semantics or interfaces of its guest operating systems, it depended heavily on understanding the ways they configured the hardware. Case-in-point: we considered supporting OS/2, a legacy operating system from IBM. However, OS/2 made extensive use of many features of the x86 architecture that we never encountered with other guest operating systems. Furthermore, the way in which these features were used made it particularly hard to virtualize. Ultimately, although we invested a significant amount of time in OS/2-specific optimizations, we ended up abandoning the effort.

All that work to develop hardware drivers and manage edge cases—and even create some fake virtual hardware that only existed in software—took time, but it paid off in a big way. The self-funded company quickly found itself making $5 million in a matter of months.

A user dives into the first version of VMware Workstation.

The software, at first, went relatively mainstream, given what it was. In a 1999 profile with USA Today, Greene (the company’s CEO) noted that the company was getting email from Buddhist monks who were fighting with one another over whether to run Linux on their computer or Windows. VMware allowed them to split the difference.

“When we first put together the business plan for VMware in 1998, we never thought Buddhist monks in Thailand would be part of our customer base,” Greene told the newspaper. “But it’s certainly intriguing to be this global.”

Of course, monks were only the start of it—if you follow VMware today, you are likely aware that Workstation is only a very small part of what that company became. VMs were highly usable in thousands of ways, as a mechanism for security and upkeep. (Want to put your custom intranet app on thousands of employee phones while still keeping it separate from those phones? Put it in a VM!)

Within a decade, Greene and Rosenblum had taken the company public, then sold it for more than $600 million. Two decades later, after a series of mergers and spinouts, it sold for a thousand times that. (More on that merger in a second.)

And this was largely before we had any of the niceties of the modern VM experience—or before VMware had much in the way of competition. Intel and AMD had not included native virtualization hardware on their chipsets until the mid-2000s. Meanwhile, Microsoft had to settle for acquiring Connectix, the only real competitor in the virtualization space, in 2003.

Parallels, the most popular virtualization app on the Mac, didn’t emerge until 2006, while open-source virtualization standbys like Xen, QEMU, VirtualBox, and Proxmox didn’t start making themselves known until the mid-2000s.

Put another way, VMware had a multi-year head start to dominate the world of virtualization. And in many ways, that reflects why the company has been such a key part of the enterprise for so long.

I imagine this is kind of a spicy take, but I think VMware’s success probably played a factor in x86 sticking around, despite Intel attempting to build a new generation of chip on a different architecture, called Itanium. Intel spent billions of dollars trying to make fetch happen, only for one company to awkwardly “fetch” it: Hewlett-Packard.

After all, VMware essentially proved that with an innovative use of hypervisors, you could work around the pain points that made x86 a weak option for virtualization—and it didn’t even have to break Intel’s security model to do so! If the existing x86 architecture was this capable, why switch?

VMware’s big innovation might have been born on an SGI computer, but it really proved a saving grace for Moore’s law.