Can Directories Rise Again? - The History of the Web

In an era where search is getting worse by the day, it's time to reconsider the humble directory. With AI hallucinations becoming a major problem and finding things a fraught and hostile experience, perhaps we can rebound in the other direction. Let's take a journey down memory lane and revisit the history of directories, and what might be their future.

The Birth of Yahoo!

It all started with Jerry Yang and David Filo, who created Yahoo! using a web browser and a list. They'd surf the web endlessly, finding websites that brought them joy or piqued their interest, and sort them into distinctive categories invented on the fly. When you visited Yahoo!, you didn't always know what to expect - it was part of the fun. You would come to the site and explore, maybe clicking from one category to another or lining up a dozen websites across your tabs. It was sometimes hard to find what you were looking for, but sometimes you happened upon a brand new discovery or ventured down a path you hadn't anticipated.

The Surfers

Eventually, the duo needed help to keep up with a web exploding into the mainstream. They hired Srinija Srinivasan to put together a team of curators, which Yahoo! referred to as "surfers." At the time, searching the web was not a great experience. Search crawlers were too easy to manipulate, and spammers had already figured out how to fill results with hidden keywords and artificial content. Results were too often low quality, as search engines tried to figure out how to rank things properly.

Surfers were a natural antidote to the problem. They were real people who could easily see through the bullshit. Yahoo! gave them a lot of leeway to organize things into collections and rank and organize results. And they were mostly up to the task. At Yahoo!'s peak, surfers were adding a thousand websites a day to the site. Each being powered by humans gave Yahoo! an edge. It made them a relatable path of discovery.

The Rise and Fall of Directories

Eventually, the copy cats came in the form of portals. This led to a decade-long war as company after company launched their own portals trying to grab their own slice of internet traffic. But unlike Yahoo!, this new brand of directory didn't want people to leave. They stuffed their site with news and suggestions and videos. But without Yahoo!'s surfers, and without an actual tether to the web, most fell soulless.

Google developed Page-rank, and made ranking search results far more accurate. By then, directories like Yahoo! had been so stripped of personality and utility that it wasn't even particularly hard to take them down. In the 2000s, search won out. Twenty-five years later, search appears doomed to repeat the mistakes that had allowed for their success.

The Resurgence of Directories

Search has handed over its voice to toneless, AI-generated results. They have made it their explicit goal to ensure that people never leave, never actually explore the web. Websites are desperately trying to game the system again, so that they might be able to catch a sliver of traffic from these low-quality results.

But there's an appetite for discovery on the web, driven by humans. This type of discovery is typically referred to as curation. But curators cull lists down to only the most essential finds. Surfers cast a wider net, and they may be better suited for the current moment.

The Future of Directories

The web is too big for one directory to work anymore. But communities on the web are embracing decentralization. It's okay for surfers to be somewhat fragmented. Instead of one directory, I think we will begin to see many. The excitement around a recent redesign of Ye Olde Blogroll makes that point.

There is something positively thrilling about a list of websites assembled by actual people for your delight. Those with the will to do that are our surfers. And thanks to decentralized tools, I think there can and will be many, operating independently from one another.

A New Era for Directories

Directories can rise again, powered by surfers and backed by a point of view. It's time to take the humble directory out of the shadows and give it the recognition it deserves. With decentralization and curation on the rise, the future of directories looks bright.