The Deep Web

[Buzzsaw]

Hi.

I have heard and seen many things about the 'deep web'. Google picked up some results but they weren't very specific.

Does it actually exist? If so, what happens there/ is stored there?

This topic greatly interests me and i was wondering if anyone could shed some light on it!

[CodingTim]

There's a wikipedia article on it; funnily enough, I was reading it recently.

From what I understood of the article:
The "deep web" is mostly made up of databases which although they may contain useful information, can't easily be indexed by [most] search engines because of the way they need to be queried, so you'd have to find the database first, and then search it manually.

Here's (http://en.wikipedia.org/wiki/Deep_web) the wikipedia article; there're some links to "deep web"-related sites

Unfortunately it doesn't seem to all be as mysterious as I imagined it to be :p

[TX_101]

I thought that "deep web" was just a term for anything on the internet that wasn't indexed by search engines ?

[Bewildebeast]

I thought that "deep web" was just a term for anything on the internet that wasn't indexed by search engines ?

Not so much "wasn't" as "can't be" (at least not with present technology). As Tim said, it's all the stuff in databases and the like which can only be accessed via a specific query or queries (eg patent number, keyword). Search engines can't index this stuff because the main way they crawl and index the web is by following hyperlinks. If it's not on a page that can be accessed via a hyperlink, it can't be indexed.

Think of it like this: I have a database of carrot-related recipes. I have (stupidly) decided to only make these recipes available on the results page of my own search engine (CarrotySearch). If someone else links to a CarrotySearch results page, Google (et al) can index the results page and follow the links on it to my recipes, and in turn index them. But if no-one's linked to a CarrotySearch results page, Google has no way of getting to the links that lead to my recipes. If it has no way of getting through to the page, it can't index it.

Edit:
Matt says:
damnit
Matt says:
if my fictional search engine was still being tested i could call it Beta Carroty