The end of the 404? Why we need to repair the internet’s crumbling infrastructure

The internet is nearly 30, and it’s starting to show signs of age. Links, the foundation of the internet, are in crisis. The pages plagued by dead links are not obscure Wikipedia entries or repositories of cat memes - 49% of hyperlinks in Supreme Court cases don’t work. During President Obama’s first term in office, 83% of PDFs across .gov domains vanished. Across the Harvard Law Review, 70% of links are defunct, a 2014 study found.

Thirty per cent of all links on the web redirect to dead 404 pages or out-of-date information. The problem is proliferating as more websites create transient pages with a half-life sometimes as low as 72 hours. Much like how previous generations faced the environmental waste caused by rapid post-industrial growth, how we address digital waste will determine whether our generation realizes the fundamental promise of the internet.

For publishers, a key pillar to establishing a sustainable revenue model is increasing the lifetime value of content. An outstanding feature piece or profile has an incredibly long shelf life, as long as the links in the story are kept up to date. In the frantic race to restore declining ad revenues, publishers have seemingly forgotten that their greatest asset is the long tail of articles that reporters and editors spend weeks to get just right. For most prominent publishers, 60% of traffic comes to articles that are more than two months old.

Since the advent of the internet, many scholars have treated link rot as a self-fulfilling prophecy, and an accepted externality of internet growth. But this is now a fixable problem. Instead of focusing solely on the creation of new content, the internet environment of the future will prioritize sustainability and renewal - and with that, editorial independence.

While technology has historically been the barrier to fixing link decay at scale, the obstacle now is incomplete knowledge - both of the economic potential of repairing broken links, and of the solutions to do it.

Previous attempts to address the problem have focused solely on manually fixing dead links, which while admirable, will never provide enough scale to address link rot meaningfully. We will never keep pace with the internet without a solution that proactively repairs links as they break. The good news is that current innovations in technology and machine learning can solve these problems, if publishers and technologists appreciate the impact of investing in a solution for ending the 404.

While the ephemeral nature of online commerce is inevitable, a trail of broken links is not. A classic case of link erosion online comes from publishers linking to retailers in shopping editorial. Products are constantly fluctuating in and out of stock and being replaced on retailer sites. This leads to a graveyard of 404 pages which are linked to from many corners of the internet.

Since the mass adoption of digital media, there has often been an inverse relationship between the tactics publishers have used to monetize, and their relationship to the reader experience. The proliferation of advertising technology has done little to replace lost print revenues, while the meteoric rise of sketchy "around the web" ads (which at some point have been implemented by almost every major global organization) has deeply eroded consumer trust by openly promoting false information. Repairing links presents a unique opportunity to simultaneously boost revenue by providing a better media product.

At Narrativ, we have restored more than one billion links in just two years. We have seen first-hand the commercial and UX impact, particularly for media entities. When pages break or are removed, Narrativ redirects users to the same content hosted elsewhere or relevant fallback destinations, preserving the integrity of a digital experience. We are proud to partner with a coalition of publishers and technologists committed to creating renewable links across the internet. Increasingly, forward-thinking companies understand that constructing a renewable web is simply good business.

Fixing broken links is essential to building a business model that supports the independence of journalism. We need a technology infrastructure that maintains the integrity of the fourth estate and its service to our democracy.

In the realm of innovation, repairing dead links often takes a backseat to advances in artificial intelligence, blockchain and IoT, among others. But all of these advances are dependent on a single variable: access to data. As the fundamental connectors of data, links are essential to providing the open and accurate information that underpins technological progress. Our future depends on repairing the internet's decaying infrastructure. Let’s end the 404.

Narrativ is a member of the World Economic Forum's 2018 class of Technology Pioneers. Explore this year's full cohort of Technology Pioneers here.