Today in Tedium: Thanksgiving, of course, was a day full of memories. You, of course, need a place to store all those memories—and for a long time, preferably in a way that makes them easy to access for many years to come. It would also help if that method was easy to carry, easy to store, and, well, was standardized. For years, when it came to pictures, that format was, of course, film. But in the mid-1990s, a digital format came along that eventually changed everything. It was compact, and it definitely wasn’t gone in a flash. Today’s Tedium talks about CompactFlash, perhaps the biggest turning point in file-storage history. — Ernie @ Tedium
Tonight’s GIF comes from a video of a guy trying to recover lost files from his CompactFlash card.
Read or listen to the key ideas from 7 Habits of Highly Effective People and 2000+ nonfiction books in 15 minutes with the Blinkist app! Download the app for free now. Available on iOS and Android.
Today’s Tedium is sponsored by Blinkist. Learn how to advertise here, too.
This GIF of the Road Runner going through a painted wall sort of explains quantum tunneling. It’s complicated. (via Giphy)
The research that enables flash memory dates all the way to the 1920s
The thing about flash memory, in one form or another is that it’s incredibly useful, but if you were to ask the average person who takes advantage of this memory—whether on their smartphone, on a USB stick, or via a memory card—how it works, they would scratch their heads.
In part, the reason for this is that the technology used behind flash memory has its roots in quantum mechanics, a topic that famously is obtuse to the regular person, but is essential to the way that our computers work today.
Flash memory as a concept relies on a concept called quantum tunneling, which is a key basically a low level form of energy transfer. To explain this at an extremely high level: At very small sizes, when a particle runs into a barrier of some kind, it can, with the right amount of energy, potentially travel through the barrier at a lower rate of energy.
To dumb it down further: If you were the size of an electron, you could walk through walls—kinda. These explanations, of course, are quite basic—here’s a better one.
This concept came about in the 1920s as part of a wave of advances in quantum mechanics during that era. In particular, the German physicist Friedrich Hund, who first posed the theory of tunneling at a molecular level in a series of papers starting in 1927. Over time, a series of researchers further improved on Hund’s findings.
These basic theories turned out to be major factors in the electronics industry, including in the creation of the transistor. Computer chips, in many ways, are designed to take advantage of the fact that quantum tunneling only happens at extremely small sizes.
(That cuts both ways, however. As IEEE Spectrum noted in a 2013 article, tunneling is having an effect on Moore’s Law, because transistors are starting to become small enough that they’re running into “leakage” problems.)
Eli Harari, the retired cofounder of the flash chip maker SanDisk, is quite aware of this research, because it informed his entire career, all the way back to his 1973 dissertation at Princeton University, which considered the relationships between aluminum oxide and silicon dioxide in holding a charge in place.
A few years later, this research would lead Harari, as a Hughes Electronics employee, to invent the first practical Floating Gate EEPROM, a form of rewritable memory that could be erased electronically, using the basic principles of quantum tunneling.
“When charges reach the floating gate either through tunneling or avalanche injection, they are entrapped and stored there, thus providing memory in the structure,” Harari stated in his patent for the process, published in 1978. “That is, the electric field induced by these charges is maintained in the transistor even after the field inducing force is removed. Erasing is achieved by removing the charges from the floating gate by reverse tunneling through the relatively thinner insulator region.”
This was a significant improvement over what was previously available; before this invention, an EEPROM would have needed to be erased using another source, like ultraviolet light.
Harari’s process provided some of the basic building blocks of flash memory, building blocks that he would eventually greatly improve upon and refine with his own company, SanDisk. But this technology still needed to be improved in a way that was commercially valuable and usable on a broad scale.
Soon CompactFlash, a memory format with Harari’s fingerprints, would help to do just that.
“The counselor didn’t really want to talk to him, but was nice enough to listen. For 20 minutes non-stop, in a very passionate and angry fashion, my father blasted away at the counselor, explaining to him how much of an opportunity he was depriving me of. At the end of that 20 minutes, the counselor said, ‘Ok. I’ll give your son a visa.’ I learned from my dad that tenacity is the key to success.”
— Sanjay Mehrotra, a fellow cofounder and the longtime CEO of SanDisk, explaining to Business Insider the impact of his father’s forceful actions in getting Mehrotra a visa from the U.S. government. He was rejected three times before his father reached the head of the consulate and painstakingly made his case. The move ultimately proved life-changing. SanDisk, originally named SunDisk and formed in 1988, was launched by three immigrant founders: Harari, from Israel; Mehrotra, from India; and Jack Yuan, from Taiwan. (SanDisk sold itself to Western Digital last year, in a $19 billion deal.)
A couple examples of SanDisk’s CompactFlash cards. (arvind grover/Flickr)
Why CompactFlash was a real turning point in the uptake of memory cards
These days, when we think of flash memory, we think of tinier than tiny MicroSD cards, which fit on a fingertip and are likely the way you store stuff on your Android phone.
But that innovation came after decades of improvement by SanDisk, among other companies. In some ways, the technology part, despite relying on concepts from quantum physics, was easy.
“I’m a device physicist,” Harari told Forbes in 2001. “I’ve known all along what flash can do.”
The hard part was creating a commercial market. But SanDisk was up to the task.
And in 1994, the company found its opening. Borrowing elements from the existing PCMCIA standard that was used early on for flash storage, while using a smaller form factor, the company launched its CompactFlash concept, which, while compact, is quite large by today’s standards.
The next year, the company had launched as a standard with the help of its own industry trade group, the CompactFlash Association. While SanDisk, which had changed its name from SunDisk by this point in response to feedback from Sun Microsystems ahead of its IPO, wasn’t the only manufacturer of CompactFlash in the end, that turned out to be a good thing, as it ensured the technology would have long-lasting industry support.
Initially, CompactFlash used a flash memory technology called NOR, which uses larger cells that can be read more quickly but are harder to delete or rewrite. Later, however, it moved to NAND technology, which uses many smaller cells, allowing for larger memory capacity and write speeds at the cost of read speeds. Today, most removable devices use NAND flash memory, while NOR is used more commonly in embedded systems or in cases where reliability is of utmost importance. NAND is notorious for wearing out, while it takes longer for NOR to do so.
The use of NOR was a key reason for its relatively large initial size, but that size provided advantages down the road, as CompactFlash gained a reputation as the fastest flash memory format in the land for much of its life.
“We are a virtual company. We can grow to several hundred million dollars in sales without significant capital investment. And we will.”
— Eli Harari, discussing in a 1994 New York Times article a major organizational advantage SanDisk had over its earlier competitors: The company was set up, initially, as a “virtual” company, in which it outsourced all of its manufacturing needs and had few employees in house. This made it possible for the company to ramp up quickly as flash memory became more popular. The company had thousands of employees at the time of its acquisition by Western Digital in 2016.
Examples of CompactFlash and CFast slots on a Canon EOS-1D X Mark II camera. (via Canon’s website)
Why CompactFlash survives today: Simply put, it’s got some pretty cool niches
During its early years, CompactFlash was intended as a memory storage format for digital photography—a medium that needed help at the time, as floppy disks wouldn’t cut it, even for consumer mediums.
Early on, CompactFlash was both a consumer and professional format, but it maintained its presence in the professional world, where its high disk speeds compared to other flash memory formats made it an asset in a world where digital mediums were more likely to be compared to physical film than floppy disks or hard drives.
It also helps that its physical dimensions—small enough for a pocket, large enough that it has a little heft and some resistance to the elements—came in handy for many camera-slingers, who are often working in some tough environments where losing a MicroSD card would be a pain.
“Solid-state CompactFlash cards are tough and generally reliable if treated well,” Andy Scott of the Dallas Morning News wrote in 2003. “We’ve even had a couple that went through the wash after being left in a photographer’s pocket, and they still work.”
Rob Galbraith, a onetime newspaper photographer himself, found a popular niche among fellow photographers by spending much time testing different kinds of cameras and CompactFlash cards for speeds, something he did up until 2012.
While the CompactFlash Association has since released the successor format XQD, which also prioritizes speed as its overarching mission, and is working on an even faster format called CFexpress, which can transmit data at as much as 8 gigabytes per second, CompactFlash in one form or another is supported by most professional cameras even today, a full 23 years after its introduction.
A CompactFlash card sticking out of a ZX Spectrum. (Staffan Vilcans/Flickr)
A fun side effect of CompactFlash’s PCMCIA-based original design mixed with its continual improvements is that it’s become a medium of choice for users of retro computers, with custom equipment often developed for machines like the Commodore Amiga or earlier models of the Apple Macintosh. While flash memory “wears out” over time, CompactFlash has many of the advantages of modern solid-state drives—i.e., no moving parts, potentially fast transfer speeds—at a lower cost and using a medium that’s still produced in the modern day. It beats using 25-year-old hard drives, that’s for sure.
“CompactFlash (CF) memory cards use a subset of the IDE command set, so making a bootable IDE-44 adapter is pretty trivial—it just links connections on the CF card to the appropriate connector on your IDE plug with no electronics necessary,” notes Dan Knight of Low End Mac, who added that Mac OS X’s use of virtual memory was a potential downside as it could hasten the wear-out of the card, as well as the somewhat inconsistent speeds of different CompactFlash card types on the market.
Earlier this year, Lazy Game Reviews host Clint Basinger highlighted the potential and some of the pitfalls of the approach in a YouTube clip in which he set up the device on a 486 PC. The good news: He got it working in DOS, after only a couple of hiccups, and he can take the flash card out and add stuff to it using a modern PC. The bad? He could only access 2 gigabytes of the 4 gigabyte card.
As these machines get really old and enthusiasts are looking for better ways to keep their old computers alive, CompactFlash might just provide the best path forward.
The fate of CompactFlash and SanDisk stands in sharp contrast to that of a contemporary competitor, SmartMedia, along with the work of that format’s corporate creator, Toshiba.
See, while Eli Harari came across the techniques that would eventually make flash memory possible, it was another man, Fujio Masuoka, who is actually credited for the invention. Masuoka made Harari’s floating gate process work much faster, allowing “flash” to live up to its name.
Masuoka, a onetime employee of Toshiba, is credited with inventing flash memory, building upon processes from Harari and others to make it a viable commercial product. Despite this, Toshiba failed to give him much in the way of corporate support.
Eventually, Masuoka left the company, and Toshiba, through its initial failure to see the value of Masuoka’s invention, let other companies, like Intel, take advantage of his idea. Toshiba, over time, got with the plot and put its focus more seriously onto flash memory, launching a memory card format of its own in 1995 called SmartMedia, which ultimately failed to have the success of CompactFlash, though Toshiba did eventually build a successful flash memory business.
Masuoka, meanwhile, sued Toshiba in 2004, complaining that they took advantage of an invention that they failed to support at the time, but failed to give him financial windfall that matched his patent rights. He asked for the Japanese equivalent of $9 million; he eventually got less than $1 million.
This had to have stung for two reasons: One, Toshiba soon was supplying chips to Apple for its iPods and iPhones, meaning the value of Masuoka’s invention was about to soar; and two, an inventor who has at least a partial claim to flash memory’s eventual success, Eli Harari, had become wildly successful due to his own work on flash memory. (To put this settlement in context: Toshiba just sold its memory business to a consortium of tech businesses that included Apple for $18 billion.)
At least Masuoka wasn’t alone in his mistreatment by Japanese patent law. The man who invented the blue LED, arguably an invention as important as flash memory, also had to sue to get a chunk of the pie. He received a much larger settlement, though even he wasn’t happy with what he eventually earned.
The two stories exposed a major problem with Japanese patent law, and led many companies to change their approach to licensing.
It’s a shameful state of affairs, and really hinges on one thing: The necessity to support innovators.
At least some of flash memory’s key figures earned something from their work.
