Out of the shadows: The changing landscape of the dark web

How things have changed. One of the rarest things in this age of big data and Big Brother is online anonymity. In 2015, New Yorker published an updated cartoon reflecting this. The dogs look older, jaded. “Remember when, on the Internet, nobody knew who you were?”

It takes extreme measures to preserve privacy today. It can take a high degree of technical expertise too. Install a niche operating system such as Tails; add the proprietary Pretty Good Privacy (PGP) encryption software; access the internet via a virtual private network (VPN); use secure email providers such as Proton, and private chat apps such as Wickr and Threema; search the web using DuckDuckGo; share files using OnionShare; do it all through a dark web browser such as Tor.

And that’s just for the computer. Staying anonymous on a smartphone is a far more intense operation.

Most people won’t go to such extreme lengths; many are happy to minimise tracking by doing just that last bit: accessing the internet via Tor, or The Onion Router.

The technology for Tor was originally developed by the US Navy, as a secure communications technology for military intelligence. The platform was opened up to the public in 2004, because it would be a poor military intelligence platform if only spies used it.

Tor remains the most widely used gateway to the dark web.

Its two key features are anonymous browsing and the anonymous hosting of websites. Roughly speaking, this is achieved by routing web traffic through multiple computers on a private network. A VPN works similarly, but Tor is decentralised, with servers now run by volunteers around the world. (Since 2006, development of the software has been overseen by a not-for-profit organisation called Tor Project.)

Tor websites end with .onion and are largely untraceable, invisible on the clear net, and are not indexed by mainstream search engines such as Google.

Operating in a similar fashion are alternatives such as Freenet and Invisible Internet Protocol (I2P), but Tor remains the most popular and is as synonymous with the dark web as Google is with internet search.

Fifty shades of dark

A dark web browser can be used to access regular websites, but a little cursory window-shopping shows that one can get almost anything illicit here. There is stolen personal data (bank account and credit-card details), fake passports and driver’s licences, ransomware, malware, narcotic drugs, illegal pornography.

There are self-help sites ranging from the criminal to the comical. How to 3D-print a gun. How to steal your neighbour’s wi-fi. How to eat for free at Taco Bell.

It’s a hall of mirrors, with no way of telling what’s real. Shopfronts promise drugs from anonymous vendors, but there is the risk that they’ll never be shipped; or that they will be shipped, and tracked, by undercover law-enforcement agents.

There is some method to the madness. Some dark web sites collect an initial security deposit from vendors; money that buyers pay is sometimes deposited in escrow accounts, to be released to vendors only after confirmation of shipping.

But the most successful marketplaces must work hard to protect themselves. Gang wars are common. Administrators of a marketplace may send extremely high volumes of automated bots to a rival, creating a digital traffic jam, crashing the site or compelling denial of service. This is why dark web sites have some of the toughest CAPTCHA tests on the internet.

Seller websites appear and disappear sometimes in a span of hours. E-shops publicise the addresses of mirror sites, in case their main site is forced down; but this opens another door to phishing attacks. Anarchy is the flip side of total anonymity. Only the paranoid can thrive.

The bright side

The law is a visible and invisible presence on the dark web. There are websites run by agencies, including the US’s Central Intelligence Agency (CIA), that invite users to alert enforcers to illegal activity. But that’s just the outermost layer of the onion.

In 2013, America’s Federal Bureau of Investigation (FBI) took down Silk Road, then the largest dark web marketplace, two years after it was set up.

AlphaBay, launched in 2014, grew to ten times the size of Silk Road. It was taken down in 2017 by FBI, in coordination with Europol and police teams in Thailand, the Netherlands, Lithuania, Canada, UK and France.

Other busts occur frequently. Cash and cryptocurrency are seized, drugs and counterfeits confiscated. New marketplaces rise in their place and it starts all over again, like a real-world game of whack-a-mole.

But there’s more to the bright side.

With authoritarianism on the rise around the world, another core function of the dark web is free speech. Media outlets such as BBC, The Guardian, The New York Times and ProPublica have .onion versions of their websites, so that readers can engage with them even if harsh curbs are in place where they live. Many of these news sites have had a presence on Tor for years, mainly as a way for whistleblowers to reach out to them.

Facebook, for similar reasons, set up a .onion version in 2014 (rather ironic, given its run-ins with privacy-rights advocates). Far-right websites such as The Daily Stormer, de-platformed on the clear net, have moved to .onion sites. So have some chan sites de-platformed for violent and hate speech. (Chan sites are loosely moderated, anonymous forums that allow users to post text and photographs.)

Meanwhile, dark web community discussion forums have been drawing growing numbers. Dread, launched in 2018 and modelled after Reddit, allows users to share posts about dark web markets anonymously but publicly. VormWeb, launched in 2020, is a search engine for onion sites. Tor.taxi and Dark.fail offer directories of different sites and anti-phishing links. Security expert Alec Muffett has published a list of mainstream and “Tor-for-good” .onion websites, including those of many civil society organisations.

The long shadow

So, how big is the dark web really? Not so big, compared to the clear net. There are 1.1 billion websites in the world and around 200 million of them are considered active. There are fewer than 60,000 .onion websites, of which only about 15% are active.

The Internet has more than 5 billion users, including 729 million in India. According to data from the Tor Project, that network has 3 million active users globally, including about 100,000 from India. The overwhelming majority (96%) of Tor traffic worldwide is aimed at the surface web; less than 4% of traffic heads to dark web sites. This suggests that only a tiny minority of users are referring to .onion sites at all.

That’s good news and bad news. One reason so little traffic is directed at the dark web sites is because one can access almost anything on the clear net.

E-commerce sites sell counterfeit goods, porn is ubiquitous, hate speech is in your face, phishing is rampant, drug dealers advertise on social media. In fact, the clear net may provide less cover, but it offers greater reach. The big difference is content moderation; most clear net sites are dutybound to try to stamp out criminality.

Similar crackdowns are now occurring on certain dark web marketplaces too. There have been reports of some spaces banning child pornography, videos of violence and ads for or by hitmen, since this increases scrutiny and puts the marketplace in the crosshairs of law-enforcement far sooner.

Why have countries not simply banned the network? China has blocked access to Tor. Russia has repeatedly tried to do the same.

Many countries keep that door open for the obvious reason (Wouldn’t you like to have a way for informers in a troubled neighbourhood to reach out to you?) Tor is likely still used widely for espionage. Of course, the optionality here is double-edged.

Meanwhile, research shows that as a regime turns more authoritarian, the use of Tor in that country goes up. It becomes a route past political repression, a common meeting ground for those under threat. Because of how hard it is to tell who’s using it, it’s hard to estimate how that number will grow.

But before the moral arc of the universe bends towards justice, it may just have to pass through dark tunnels of Tor.

(Kashyap Kompella is an industry analyst and a visiting professor for artificial intelligence at the BITS School of Management [BITSoM])