Deep Web: The Hidden History of Everyday Online Content
An inbox, an online banking page, and a subscription magazine can all sit beyond the reach of an ordinary search engine. They belong to the Deep Web because their contents are not indexed by standard search programs. Access may require a password, registration, or payment, even when a direct address leads to the service. The surface web, by contrast, is accessible to anyone using the internet.
The vocabulary began with attempts to describe what search engines missed. In 1994, Jill Ellsworth used Invisible Web for websites that had not been registered with a search engine. Frank Garcia discussed similar hidden sites in January 1996. Bruce Mount and Matthew B. Koll of Personal Library Software also used Invisible Web in a December 1996 press release about a search program. Computer scientist Michael K. Bergman's 2001 study brought the specific term Deep Web into accepted use. Mount and Koll's terminology referred to their AT1 search program.
The terminology became tangled in 2009, when discussions of search terminology appeared alongside illegal activity on Freenet and the darknet. Later coverage of the black-market website Silk Road encouraged use of the broader term for dark web activity. Reporters Kim Zetter and Andy Greenberg recommended keeping the terms separate. The wider category concerns material a traditional search engine cannot access, while the narrower category concerns sites hidden intentionally and unavailable through standard browsers and methods.
For ordinary services, the obstacle to indexing often comes from access controls. Web mail, cloud storage, and online banking may require a password or other security information to access their content. Some social media pages and profiles also have restricted access, and some forums require registration before their contents can be viewed. Paywalled video services, magazines, and newspapers add another kind of barrier. Deep Web material can therefore be available to its intended users through a direct URL or IP address while remaining outside a search index. Tor also allows anonymous access to .onion websites while hiding a user's IP address.
Other barriers arise from the way a site generates its pages. Some pages appear only after someone submits a form or queries a database. A crawler following hyperlinks may never encounter those results, because the number of possible queries is indeterminate. Content may also change with a visitor's IP address or previous navigation. Scripts can produce links or download material dynamically, and text inside certain images, videos, or file formats may go unrecognized by a search program. Pages without links from other pages can escape crawling too. Even on crawled pages, search engines do not always detect every backlink.
Archives add a different kind of invisibility. The Wayback Machine allows visitors to view older versions of pages, including sites that have disappeared or become inaccessible. Those past versions cannot be viewed through a search of the current site in the same way as its present pages. Meanwhile, some sites deliberately restrict crawlers through robots rules, CAPTCHAs, or instructions preventing cached copies. They may offer their own internal search engines instead. The Deep Web thus includes both information preserved from an earlier time and information whose current owner controls how automated programs explore it. The reason for being unindexed varies with the service.
Researchers developed tools to reach material behind forms. In 2001, Sriram Raghavan and Hector Garcia-Molina at Stanford presented a crawler design that used terms from users or query interfaces to submit forms and explore hidden content. Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho at UCLA built a crawler that automatically generated meaningful search queries. The University of Utah's DeepPeep project gathered forms in different subject areas using focused crawling techniques, with National Science Foundation sponsorship. Other specialized search services included Intute and Scirus. Intute ran out of funding and became a temporary static archive in July 2011; Scirus retired near the end of January 2014.
Commercial search engines also sought ways into the Deep Web. Google introduced the Sitemap Protocol in 2005, allowing servers to advertise accessible URLs and help automated programs discover pages without ordinary incoming links. Google's form-surfacing system calculated submissions and added the resulting pages to its index. It selected keyword values for text fields, identified fields requiring particular kinds of values such as dates, and chose a limited number of input combinations suitable for indexing. A separate approach arrived in 2008, when Aaron Swartz designed Tor2web, a proxy that let common browsers access hidden services whose addresses ended in .onion.