Terms of use: Peeling back the layers of the dark web

Data scientists like to describe the vast expanse of the world wide web as a large iceberg. The surface web or clear net is the visible tip and consists of easily accessible sites indexed by search engines such as Google and Bing. This makes up about 4% of the content on the internet.

PREMIUM

Dive below the surface and an overwhelming majority, 96% of content, sits on the deep web. These are restricted-access sites. The dark web is the deepest sliver of the iceberg, constituting less than 0.01% of the content on the internet. It can be accessed only through special software. Take a look at how these stand apart.

Surface Web: This is the visible web that we all know and use. It is indexed by standardised web search engines and accessed through numerous free-to-install browsers.

Deep Web: Also called the invisible web or hidden web, these are the parts of the internet that exist beyond the reach of mainstream users and standard search engines. Government and corporate databases, company intranet systems, academic journals, legal documents and paid content are among the types of material stored here. Access involves making it past a paywall or logging in with a password.

Dark Web: Only a fraction (about 0.01%, according to Encyclopaedia Britannica) of the deep web qualifies as the dark web. This is a part of the internet that can only be accessed through special software such as The Onion Router (or Tor), Freenet, and Invisible Internet Protocol (I2P). All IP addresses are encrypted; all transactions are anonymous. Surveillance is difficult because the network is decentralised, messages passing between servers run anonymously by volunteers around the world.

Dark net vs clear net: The unencrypted networks that most of us use to access the surface web are referred to as the clear net. These host the .com, .net, .org, .in and other domains that can be accessed via any web browser. Darknet is the term for the encrypted networks that are used to access the dark web. Each such network — whether Tor, Freenet or I2P — must be accessed via a separate software program.

Who, what, when: A timeline of the dark web

1995: At the US Naval Research Laboratory, a new, secure mode of intelligence transmission is developed by computer scientists Michael G Reed and David Goldschlag, along with mathematician Paul Syverson. They call it onion routing. It allows messages to be covered in layers of encryption, and transmitted through a series of off-grid servers.

1999: Ian Clarke, an Irish programmer studying at the University of Edinburgh, creates a revolutionary, decentralised information storage and retrieval software for his thesis project. It allows people to share files, chat online and browse websites in total anonymity. He calls it Freenet and releases it to the public in 2000. It works by storing encrypted snippets of data on the hard drives of users across its network.

2002: The Onion Router (TOR) is opened up to the public, with some help from computer scientists Roger Dingledine and Nick Mathewson of the Massachusetts Institute of Technology (MIT). It can’t be much help to US intelligence if its only users are spies, so the public is invited in to make up a general population.

2003: Amid concerns over privacy and surveillance, new platforms such as the Invisible Internet Project (I2P) are born. The idea comes from cyber security expert Lance James; most developers remain anonymous. I2P routes data through a volunteer-maintained network of devices around the world, thus scattering traffic in a way that makes it difficult to intercept.

2006: Tor Project, a not-for-profit research education organisation, is founded to oversee development of the software.

2009: Two big developments — the launch of Bitcoin, the world’s first cryptocurrency, and the creation of the Tor browser — change how the darknet is accessed. The term dark web begins to appear in newspaper articles about the growing use of the Tor network, via the Tor browser, for illicit and illegal activities. Drugs, weapons, pornography and stolen data are among the products and services being traded even more invisibly now, thanks to cryptocurrency.

2010: Tor becomes part of a growing pro-democracy revolution that begins with the Arab Spring in 2010, through the Syrian civil war (2012-). To circumvent censorship and surveillance, videos of police firings, bombings and other atrocities are released here, to find their way to mainstream clear net platforms such as Twitter and YouTube.

2013: America’s Federal Bureau of Investigation (FBI) takes down Silk Road, then the largest dark web marketplace, two years after it was set up. American entrepreneur Ross Ulbricht, founder of the marketplace, is arrested and convicted on charges of drug trafficking, among others, and sentenced to life in prison.

Computer intelligence consultant-turned-whistleblower Edward Snowden uses Tor to leak highly classified information relating to overseas operations — including numerous surveillance programmes — conducted by the US National Security Agency. “For me personally, Tor was a life changer, bringing me back to the Internet of my childhood by giving me just the slightest taste of freedom from being observed,” he would later write in his autobiography, Permanent Record (2019).

2017: The New York Times sets up a mirror site on Tor, so that it can be accessed in regions that have blocked, censored or otherwise disrupted free speech online. BBC makes its international website available via Tor in 2019, citing blocks in countries such as China and Iran. The Guardian has used Tor to communicate with whistleblowers since 2014 and establishes a .onion presence in 2022.

The dark web marketplace AlphaBay, launched in 2014, has grown to ten times the size of Silk Road and is taken down by the FBI, in coordination with Europol and police teams in Thailand, the Netherlands, Lithuania, Canada, UK and France. Alleged founder Alexandre Cazes is arrested but found dead days later in a Thai jail.

Busts have been carried out periodically since then, and a number of marketplaces have begun to ban child pornography, videos of violence and ads for or by hitmen, since this puts them in the crosshairs of law-enforcement far sooner.