The Technorati Generation

The search engine that kept up at the speed of blogging created a “search war” with Google. But eventually, the speed of blogging just wasn’t fast enough.

Today in Tedium: The thing that we often say about our modern social media ecosystem is that we’re often playing within the walled gardens of others. This is largely by design—Facebook, or Twitter, or Snapchat, or TikTok have more control over the final result if they’re the ones hitting the switches back at HQ. But there was once a time that all that chatter, albeit at a smaller scale, was happening on disparate platforms that nobody owned. There was no walled garden—only a whole lot of weeds, growing freely around the internet. Someone had to make sense of it, and that led to the rise of the blog search engine—a concept popularized by Technorati. (Remember that?) Today’s Tedium considers the forgotten period when blog search engines were a big deal. — Ernie @ Tedium

Thanks to Simon Owens for the idea here. He has a Substack, if you’re curious.

Find your next favorite newsletter with The Sample

Each morning, The Sample sends you one article from a random blog or newsletter that matches up with your interests. When you get one you like, you can subscribe to the writer with one click. Sign up over this way.

Today’s Tedium is sponsored by The Sample. (See yourself here?)

Technorati, as the site appeared in 2007. (buou/Flickr)

The early blogosphere was defined by side projects that became businesses. Technorati was one of those.

When it comes down to it, blogging has always been shorthand for a way to describe conversational writing on the internet, though the specific terminology took a while to find its footing. Some early blogging notables, like late science-fiction author and Byte columnist Jerry Pournelle, were slow to embrace the term.

But it ultimately took off, in no small part thanks to a series of small startups that were generally passionate about the idea. Perhaps the most famous was Blogger, a platform notably created by Evan Williams and Meg Hourihan in 1999 under the Pyra Labs brand. Hourihan notably developed the Kinja blogging platform used by Gawker Media-descendant sites even today; Williams, of course, is a cofounder of Twitter and Medium.

Small startups like Pyra Labs were the lifeblood of early blogging, and played a fundamental role in helping to set the parameters of early internet culture as it evolved from the excesses of Web 1.0 (think Kozmo and Pets.com) into what became known as Web 2.0. South by Southwest’s interactive portion gained much of its modern cultural cachet in large part because Williams and other bloggers decided to embrace it.

It was not a particularly stable time for the web, admittedly. Infamously, Pyra Labs, which did not have a business model for Blogger as the company was intended to develop project management software, saw its staff members leave one-by-one as its funding dried up, leaving Williams as its only employee.

And years later, many of the ideas invented around this time got swallowed up into the walled gardens of the startups and corporate giants that followed these digital pioneers.

One of those digital pioneers was a guy named David Sifry. A blogger himself, he wanted a way to better understand how his own site was doing on the internet. So he built a way to do that. Somewhat similarly to Pyra Labs, the tool that he built, which he called Technorati, was not necessarily intended to be a business. He already had a startup that he was working at, which focused on wireless computing.

“Essentially, it is a site that creates a set of web services that I’ve always wanted for myself—services layered on top of the wealth of current search functionality and tools available for bloggers,” he wrote in a 2002 blog post.

Among those features were a number of basic tools that told users who was writing about or linking to a blog post, how your site rates in relation to specific Google search terms, historical information on how sites are ranking over time, and—most importantly—where everyone ranked.

The ranking element effectively made it a search engine for blogs, a great way to find new voices at a time when the “blogosphere” was a new and vibrant thing. And rather than grabbing this data only occasionally, as many search engines did at the time, it was built around the pulse of the digital ecosystem. Effectively, it filled an immediacy gap for blogs that early search engines were not designed for.

But the data it offered bloggers might have been the most important part of its reason for being.

These days, companies charge hundreds or thousands of dollars a month for the kinds of analytics Technorati generated. Sifry was charging $5 per year for access to historical information on Technorati. This was before we had Google Analytics or anything similar, so this added data was a big deal.

At first, Technorati was a side project for Sifry, just like Blogger was for Pyra Labs. But the cultural tide quickly changed things and made Technorati hugely valuable to large companies. In its first year online, it wasn’t just small-scale bloggers that were interested in the data Sifry had gathered, but major news outlets such as The New York Times and Reuters. As Inc. reported in 2006, Sifry was soon signing a deal to license his technology to AOL—and quitting his job to make Technorati a full-time thing.

Technorati swag, circa 2005. (Nikita Kashner/Flickr)

And, just like blogging itself, Technorati grew fast. To give you an idea, the company scored an impressive partnership with CNN tied to the 2004 Democratic National Convention. Sifry found himself blogging and reporting for CNN, rounding up a variety of voices that Technorati was well-positioned to highlight.

“I hope that this helps people in the living room realize that they can be participants,” Sifry said, according to PR Week.

Not bad for a company that less than two years before was a weekend project for Sifry.

Blogger and Technorati were two of the earliest Web 2.0 tools, predating Friendster, Facebook, MySpace, Digg, Reddit, and numerous other mid-2000s icons of blogging and social media.

At the center of this trend was the concept of the blog, which encouraged people to own their own piece of the internet. Many of the later Web 2.0 tools focused on trying to get people to spend as much time as possible within their URL; Technorati, meanwhile, was really focused on helping this universe of bloggers become better at blogging.

Other services that mined this territory, like Popdex and the Mark Cuban-backed IceRocket, played an important role in the rise of the blog-targeted search engine, whether by ranking the most popular sites or by making it easier to track information in real time. (After all, new posts from amateur pundits were always filtering in.) But none cast a shadow quite as large as that of Technorati, whose ranking system meant something for bloggers. To appear on the list, particularly in a high spot, was a sign of validation. Yes, you were good enough at blogging.

It was only a matter of time before bigger players intruded on Technorati’s territory. And yes, when I say “bigger players,” I mean Google.

five

The number of months it took the blogosphere to double in size, according to the 2005 “State of the Blogosphere” report completed by Sifry, an annual report that tracked growth in the space. The annual reports, once watched as closely as Mary Meeker trend reports, became increasingly sophisticated over time; the 2008 version of the report, for example, went into great depth about how much money the average blogger could make from the practice, and further tracked a ballooning trend that at that time represented 1.2 million bloggers that had just signed up for Technorati’s service. (Many more blogs existed than just that at the time.)

Google Blog Search had a lot of Google News energy. (via Alia Al Seeri)

For a brief moment, blog search was a big deal. Until it wasn’t.

In as many ways as one company can, Google has spent much of the past two decades taking control of many of the general elements of running an independent website. For example, Google Analytics, which evolved from the analytics service Urchin, has long been a de facto standard for tracking website data.

And when the RSS feed gained popularity during the first half of the 2000s, Google launched one service, Google Reader, and acquired another, FeedBurner.

But the company is not known for nostalgia, and will remove support when it’s clear a trend has run its course—or Google wants that trend dead. And blogging, while not dead, is a trend the company has clearly moved past. Blogger (acquired in 2003), while still active, languishes, only receiving occasional notice from Google despite being one of the largest sites on the internet. So, too, does FeedBurner. And it famously killed Reader outright.

Such is the story of Google Blog Search, the company’s response to the rise of Technorati. It was there when blogging was a hot trend and gone by the time the trend had started to fade. Launched in 2005, it allowed users to search for individual blog posts, both by relevance (to see the most contextually relevant search results) and by date (to see the latest). It was faster than other options, but with fewer bells and whistles. Early bloggers were skeptical. In a 2005 post on TechCrunch, founder Michael Arrington noted that other tools, particularly Technorati, had more detailed search functionality, and that post relevance wasn’t as good as other options.

“Overall, significant room for improvement exists,” Arrington wrote.

Ultimately, the competition between Technorati and Google was a good thing, no matter who was actually “better” at the job. A 2007 Search Engine Journal blog post noted that the two sites ultimately complemented one another by touching on different technological aspects. While Google was better at a more fine-toothed approach by date, Technorati ultimately brought up better results and had more filtering capabilities.

Other companies, including Microsoft and Yahoo also dived into the space at the time.

For his part, Sifry, who remained CEO of Technorati until 2007, chose to stay the course in the face of the new competition, telling Inc. in 2006, that Google’s competitor was “solid but simple.” Nonetheless, the entry of Google into the space ensured an uphill battle for the little blogging search tool that could.

In the Inc. piece, a number of tech and marketing figures were asked to comment on Technorati’s then-current business model, and the critics seemed less than favorable, suggesting that it wasn’t focused enough, or that the company should focus on an acquisition. One expert quoted, Engadget cofounder Peter Rojas, admitted he was a heavy user of the site, even as he suggested it was not going to win the battle for blog search supremacy.

“Technorati’s strategy is not unique, so its execution has to be perfect,” Rojas stated. “Sooner or later, somebody’s going to get it right. I hate to say it, but it’s probably Google.”

In a way, Google did get it right by not overplaying its hand—creating an effective response to sites like Technorati but not letting them overwhelm their model. Blog Search was just good enough that you didn’t really need to go to another website to find useful results. But it didn’t give the blog outsized influence on its overall results, which was probably the right move, considering the fact that the idea of a blog search engine quickly fell out of date. By the time Twitter gained a footing in the mainstream in 2009—six short years after Ev Williams sold Blogger to Google—searching for things that people said in real-time was no longer a differentiator. It was table stakes.

(Perhaps reflecting this, Google Blog Search was effectively retired in 2011 and exists quietly as a search feature within Google News.)

Technorati tried building an ad business for bloggers starting in 2008. It eventually usurped the rest of the business. (via ProBlogger)

And that was bad news for Technorati, which eventually evolved into more of a services business, with website widgets and advertisements bringing elements of the Technorati experience across the web—even if it dampened the experience of the Technorati website. “Today, only a small percentage of Technorati’s total network traffic of 25 million U.S. unique visitors per month actually visit Technorati.com,” Michael Arrington wrote in TechCrunch in 2009.

As I mentioned earlier, walled gardens won the internet, and Technorati wasn’t a walled garden. It was also too small to win by sheer scale, like Google could.

Sure, its list of top blogs was considered definitive, but was that enough build a company around? In the end, the answer was no.

“Our new website becomes the entry point to our advertising platform, our core product. There we proudly stand beside website publishers by providing them the right toolset to navigate and monetize the ever-evolving online advertising marketplace.”

— A passage from a 2014 Technorati blog post, announcing a decision by the company to stop focusing on its ranking algorithm in favor of its advertising platform. (Disclosure: I at one time time used their platform for advertising when running my former site.) This was a significant change for the platform and perhaps reflected the state of the blogosphere in 2014, no report needed. In some ways blogging had become so overwhelmingly commonplace as to become meaningless to track in the way Technorati had in the prior decade-plus; in others, it was an outmoded way of thinking about the practice. By 2016, Technorati had sold to the technology services company Synacor for $3 million—a fraction of the money it raised in its history and now remains online as, effectively, a generic web portal.

As a blogger during the latter years of this period, it cannot be overstated enough how many little startups were trying to win people over, to put this widget or that widget on their sites.

It became way too easy to clutter things up in an effort to find just the right tool that could help you squeeze a little more traffic or revenue out of the lemon that was your blog. (Less considered, of course, was that all of these widgets were siphoning data about your readers in exchange for this little feature or that neat doodad.)

Technorati was in many ways the first shot in that general direction, though it was not the only one and certainly not the worst offender. But the purity of its original model, which was lost to time and ever-quickening innovation, was valuable for what it was. Rather than forcing us to live in someone else’s world, it tried to make sense of the quickly growing blogosphere, while allowing that sphere to live in place elsewhere.

By making the world of blogs trackable in real time, it made them more communal—a feeling some of us are trying to get back to today. But the globe kept spinning too fast for any one site to keep up—even one designed for that purpose.

Pete Cashmore, the founder of Mashable, which at the time was an important blog focused on the then-burgeoning growth of social media, was declaring Technorati dead at the hands of Twitter as early as June of 2008.

“Yes, Google Blog Search was a harsh blow for the once prominent blog search engine, but Technorati’s greater failing was to turn a blind eye when the breaking news conversation moved to a different medium: microblogging,” he wrote.

Perhaps 2008 was a bit early to be making that call, but certainly Cashmore had a point—one that was ultimately proven right. By 2010, Technorati was showing signs of struggle, with some users reporting site approvals slowing down to a crawl. Perhaps it was a harbinger. Four years later, the company would be sunsetting the main reason the platform existed.

The reason people cared about Technorati was because it could pick up on conversations, fast. But when social media became faster than blogging, a dedicated blog search engine no longer made sense.

But it was nice while we had it.

--

Find this one an interesting read? Share it with a pal!

Are you a Mastodon user? Be sure to give Ernie a follow!

And thanks to The Sample for giving us a push this time. Be sure to subscribe to get an AI-driven newsletter in your inbox.