The other 96%.

Imagine a world that is composed of multiple realities, with realities being subjected to every person who dares to be an observant. A world where multiple realities collide, collapse and hide from each other, leaving only a small portion of themselves accessible to the general public, the proletariat, and largely remaining hidden, becoming the sole possession of the ultra powerful, who then use it to change the reality itself. Ladies and gentlemen, that’s the Internet for you!

I can’t vouch for the veracity of what I’m about to write because this is a pretty dark and recondite phenomenon with no clear authority in terms of both information and control. However, some studies on this subject matter have been published and verified by experts who have experiences from this astounding world. So, let’s plunge into what is now formally known as the “Deep Web”, the dark and sometimes grisly counterpart of the morsel Internet as we normally witness on a day to day basis.

It is believed that once we witness something again and again, then courtesy of the reinforcement, that thing in itself becomes our own version of the reality, We never bother to touch what’s beyond what we witness. Time constraints and lack of knowledge blindfold us, and as a result, we can never really find out how constricted our version of the universe is. I was absolutely stupefied when I was finally able to take my own blindfold off. We surf the Internet for hours every day, roaming through content of all types, running queries on search engines, and finally feeling content on finding all sorts of documents and information we ever sought. And that is it. Truth is, there is an entirely different realm of the Internet that actually commences from the very point our infatuation with it ends. This is the real Internet as true patrons of the same would call it. And this is the Internet that you and me, didn’t have access too. Well only until you read this. So, you ready?

Internet is a complex network of networks. There is panoply of routers, hubs, switches and layers and layers of communication protocols that run on these devices to formally form what we fondly call and amateurishly believe to be the "Real Internet". In reality, only 4% of the actual Internet is accessible to us, with “the other 96%” escaping our eye. Truth be told, our version of the Internet is all that we can access either through search engine queries and various links and IP address file system paths that may be shared with us. The gigantic search engines with all their finesse are able to crawl and index multiple types of web content across the globe and make a formidable database of it which they bring at our disposal by providing us query and retrieval search mechanisms. While you are usually stunned by Google’s amazing splendour of producing zillions of search results in nanoseconds, it is still a meagre portion of all that is out there. This 4% of the Internet is called the Surface or Indexed Internet. This is easily accessible to all Internet users. However, a real gateway guards these minion search engines from moving beyond that. The other 96% is largely comprised of the following.

The “touch me not” Internet

While website admins and bloggers may consider this a baloney to be called a revelation, normal surfers of the Internet wouldn’t know that a substantial portion of the Internet doesn’t want itself to be seen by the search engines. One always has the option of changing the robots.xml and sitemap.xml configuration files on one’s web servers to forbid access to web crawlers and hence even though these sites may still be available for public use, you never find them because you don’t have a search engine to lead you to them. The only way these sites thrive is through private and in-person link sharing mechanisms. One may believe it’s fatuous to not render oneself visible to these prolific search engines who take your page to the world. However, you wouldn’t actually want to do that if you were running say a political activism blog or something and only wanted selected patrons to access the site. Moreover, if you wanted to run a secret site which could have been vulnerable if allowed to be indexed by various search engines, you would use your own private servers to host the site. A user then would have to key in the specific IP address to access the secret site and the rest of the Internet would never even know you existed unless they bumped in by serendipity by keying the right address by chance. Various corporations, institutions, organizations and hold on to your hats -terrorist outfits; use these configurations to host data for only the privy to access. I’m damn sure you too would have witnessed this while filling up some online academic form or something which was prevented from being indexed because the site admins would have wanted a limited load on the servers. They could have ensured that by using controlled dissemination of the url.

The private and virtual private networks

This is a pretty common phenomenon in corporations which have got multiple access points to access common databases and company networks. They’d set up networks which prevent access to the outside world and forbid incoming access from the outside world as well. Hacking or rather ‘cracking’ is the art of breaking into these networks by breaching the security firewalls that are vested with the responsibility of protecting access privileges to these networks. So internal company mail and access portals are now accessible only within the company network because there are virtual tunnels which are impregnable and all company network communication, even though while it is being routed through the Internet, is actually invisible to the outside world.

The dynamic Internet

While search engines are phenomenal at accessing the static content, what about those websites which render content on the basis of information that a user fills in? So for example, a registration website which has a homepage and an about page will have these two pages indexed by the Internet, however when it comes to the registration page itself, the search engines will only be able to index the basic page where the details are to be entered. Post that, there could be a multitude of combinations regarding what more Internet forms are opened up and what new content pages are displayed. It is all dependent on what data you entered and what information is stored on the database. Hence an examination result can never be indexed by the search engines because they simply don’t know what to enter, maybe a roll number and email or date of birth combination. Just imagine the amount of data on such a site that is actually rendered only post the search engine has been blocked access. Actually, all that Internet is hidden in databases, petabytes and petabytes of data, which comes out and manifests itself on the Internet only when a user commands it to. Hence this Internet too is not accessible to the general public. You need a designated user or a ‘cracker’ to access it.

The Dark or ‘Dead’ Internet

Those who know of the origins of the Internet, know that ARPANET was the actual predecessor of multiple small scale networks across US DoD establishments and university departments, and all of them culminated into the Internet. However, so emphatic was the evolution of the Internet and so rapid was the metamorphosis, that all protocols were designed keeping only the new and future machines in mind and all the legacy systems were tantamount to codswallop on the new Internet. Hence even though these systems had an Internet of their own, that was rendered obsolete and the latest machines were never granted access to that tenebrous Internet. As you might have guessed, these old machines too were never granted access to the new Internet. Hence their Internet in all its essence, remained a cooped up phenomenon. However, it is believed that this dead space is still functional somehow in short ranges and private networks. Hence these machines wherever they exist, can still communicate with each other till their archaic protocols don’t muddle with modern devices and get blocked. Surprisingly, this is the safest of all networks today because no one knows where it exists and how to get in because even the literature on their protocol design is not available to the public at large.

And finally, the “Deep Web”

This surely is the most fascinating of all that we’re likely to stumble upon as a part of this discussion. There was a need of extreme anonymity within the realms of the Internet. Who doesn't want anyone snooping and eavesdropping on their conversations like the Thought Police does, and who wouldn't want a private space of their own, on the brilliant communication medium - The Internet itself. That is what consummated in this arcane and yet immaculate concept. Some brilliant minds came up with new communication protocols and TOR is ostensibly the accepted standard. In fact Tor is your ticket to the world of deep web! The deep web is that portion of the Internet which has a communication language of its own. In  appearance it maybe similar to ‘our” Internet but the very difference in language renders it inaccessible to us.

Tor comes up with its own browser and routing mechanisms and every network packet that is sent across the deep web, is actually routed through what they call TOR relays. Hence contrary to what happens in normal Internet communication where a source computers puts its IP address and destination IP address in network communication packets and then these packets are vehemently delivered. In Tor, the packets are bounced off through a network of about 5000 random systems or Tor relays and hence both the source and destination IP addresses are masqueraded deep within multiple layers of encryption making it nearly impossible for your next door neighborhood virtual espionage to track any of this communication to you or the recipient. Simply said, unlike conventional or our Internet, Tor makes sure that Deep Web packets remain invisible and even though the final output is still the same i.e a packet getting delivered, no one knows where it came from and where it went to.

Tor stands for The Onion Router because these Deep Web websites are actually provided with a .onion domain suffix. This ergo provides Tor and Deep Web with a new communication paradigm of their own, completely invisible to the outside world. What is this Deep Web used for? Well, since its effectiveness in ensuring one’s anonymity cant be gainsaid, it’s the favorite of those who seek it. Political activists who fear death in the event of their posts getting traced back to them, now have a resort. Personal conversations between prominent leaders is now more secure. On the flip side, insider trading can never be surfaced and you can also have ammunition, guns, drugs and what not delivered at your home! The Deep Web follows a similar design. There are blogs like this one, there are shopping portals and there are forums. In case you thought you caught me on money transfers, well Deep Web supports bitcoins! In fact bitcoins are the standard currency in Deep Web and Deep Web in itself becomes a large market or a virtual economy for the regulation of the same.

I’m not an expert on this subject but I simply wanted to give you a flavor of the Internet you never thought existed. Just like the theme of the blog that says “What you can’t see”, modern technology and its revelations have made sure that reality is subjective and purely in the eyes of the beholder. All that we see, so profound and so comported, and yet there is a reality, wading in parallel, striving and thriving at the same time, waiting for you to pay a visit. Will you?