Post Views:
0

CHAPTER
ONE

1.0     INTRODUCTION

The volume of information on the web is
already vast and is increasing at a very fast rate according to Deepweb.com [1].
The Deep Web is a vast repository of web pages, usually generated by
database-driven websites, that are available to web users yet hidden from
traditional search engines. The computer program that searches the Internet for newly
accessible information to be added to the index examined by a standard search
tool search engine [2] used by these search engines to crawl the web cannot reach most of the
pages created on-the-fly in dynamic sites such as e-commerce, news and major
content sites, Deepweb.com [1].

According to a study by Bright Planet [3],
the deep web is estimated to be up to 550 times larger than the ‘surface web’
accessible through traditional search engines and over 200,000 database-driven
websites are affected (i.e. accessible through traditional search engines).
Sherman & Price [4], estimates the amount of quality pages in the deep web
to be 3 to 4 times more than those pages accessible through search engine like
Google, About, Yahoo, etc. While the actual figures are debatable, it made it clear
that the deep web is far bigger than the surface web, and is growing at a much
faster pace, Deepweb.com [1].

In a simplified description, the web
consists of these two parts: the surface Web and the deep Web
(invisible Web or hidden Web) but the deep Web came into public awareness only
recently with the publication of the landmark book by Sherman & Price [4],
“The invisible Web: Uncovering Information Sources Search Engines Can’t See”.
Since then, many books, papers and websites have emerged to help further
explore this vast landscape and these needs to be brought to your notice too.

1. Statement of Problem

Most people access Web contents with Surface Search
Engines and 99% of Web content is not accessible through Surface Search Engines.

A complete approach to conducting
research on the Web incorporates using surface search engines and deep web
databases. However, most users of the Internet are skilled in at least
elementary use of search engines but the skill in accessing the deep web is
limited to a much smaller population. It is desirable
for most user of the Web to be enabled to access most of the Web content.  This work therefore seeks to
address problems such as how Deep Web affects: search engines, websites,
searchers and proffered solutions.

1. Objective of the study

The broad objective of this study is
meant to aid IT researchers in finding quality information in less time. The
main objective of the project work can be stated more clearly as follows:

1. To describe the
   Deep Web and Surface Web
2. To compare deep
   web and surface web
3. To develop a
   piece of software to implement a Deep Web search technique

1. Significance of the study

The study on deep web is necessary
because, it brings to focus problems encountered by search engines, websites
and searchers. More importantly, the study will provide information on the
results of searches made using both surface search engines and deep web search
tools. Finally, it presents deep web not only as a substitute for surface
search engines, but as a complement to a complete search approach that is
highly relevant to the academia and the general public.

1. Literature review

What is Deep Web?

Wikipedia [5], defined the surface Web (also known as the visible Web or indexable Web) as that portion of the World Wide Web that is indexed by conventional search engines. Search engines construct a database of the Web by using programs called spiders or Web crawlers that begin with a list of known Web pages. For each page the spider knows of it retrieves the page and indexes it. Any hyperlinks to new pages are added to the list of pages to be crawled. Eventually all reachable pages are indexed, unless the spider runs out of time or disk space. The collection of reachable pages defines the surface Web.

For various reasons (e.g., the Robots Exclusion Standard, links generated by JavaScript and Flash, password-protection) some pages cannot be reached by the spider. These ‘invisible’ pages are referred to as the Deep Web.

Bergman [6], defined the deep Web (also known as: Deepnet, invisible Web or hidden Web) to mean World Wide Web content that is not part of the surface Web indexed by search engines. Dr. Jill Ellsworth coined the term “Invisible Web” in 1994 to refer to websites that are not registered with any search engine.

Sherman and Price [4], defined deep web as text pages, files, or other
often high-quality authoritative information available via the World Wide Web
that general-purpose search engines cannot, due to technical limitations, or
will not, due to deliberate choice, add to their indices of Web pages.
Sometimes referred to as invisible web” or “dark matter”

          Origin of Deep Web

In 1994, Dr. Jill H. Ellsworth a university professor who is also an
Internet consultant for Fortune 500 companies was
the first to coin the term “Invisible Web” [6]. In a January 1996 article, Ellsworth states: “It would be a site that’s possibly reasonably designed, but they didn’t
bother to register it with any of the search engines. So, no one can find them!
You’re hidden. He called that the invisible Web”.

The first commercial deep Web tool (although they referred to it as the “Invisible Web”)
was AT1 (@1) from Personal Library Software (PLS), announced December 12^th,
1996 in partnership with large content providers. According to a December 12^th,
1996 press release, AT1 started with 5.7 terabytes of content which was
estimated to be 30 times the size of the nascent World Wide Web.

Another early use of the term
“invisible web” was by Bruce Mount (Director of Product Development) and Dr.
Matthew B. Koll (CEO/Founder) of Personal Library Software (PLS) when
describing AT1 (@1) to the public. PLS was acquired by AOL in 1998 and AT1 (@1)
was abandoned [7], [8].

AT1 is an invisible web which allows users to find content “below,
behind and inside the Net” therefore users can now identify high quality
content amidst multiple terabytes of data on the AT1 Invisible Web; top
publishers join as charter members.

1.  The Internet and the Visible Web

The primary focus of this project work
is on the Web and more specifically, the parts of the Web that search engines
can’t see (known as the invisible Web) but in order to fully understand the
phenomenon called the Invisible Web, it is important to first understand the
fundamental differences between the Internet and the Web.

Most people tend to use the words “Internet”
and “Web” interchangeably, but they are not synonyms. The Internet
is a networking protocol (set of rules) that allows computers of all types to
connect to and communicate with other computers on the Internet. The Internet’s
origin traced back to a project sponsored by the U.S. Defense Advanced Research
Agency (DARPA) in 1969 as a means for researchers and defense contractors to
share information. The World Wide Web (Web), on the other hand, is a
software protocol that allows users to easily access files stored on the
Internet computers. The Web was created in 1990 by Tim Berners-Lee, a computer
programmer working for the European Organization for Nuclear Research (CERN).
Prior to the Web, accessing files on the Internet was a challenging task,
requiring specialized knowledge and skills. The Web made it easy to retrieve a
wide variety of files, including text, images, audio, and video by the simple
mechanism of clicking a hypertext link. Hypertext is a system that allows
computerized objects (text, images, sounds, etc.) to be linked together, while
a Hypertext link points to a specific object, or a specific place with a text;
clicking the link opens the file associated with the object [4].

The Internet is a massive network of networks, a networking infrastructure. It connects millions of computers together globally, forming a network in which any computer can communicate with any other computer as long as they are both connected to the Internet. Information that travels over the Internet does so via a variety of languages known as protocols.

The World Wide Web, or simply Web, is a way of accessing information over the medium of the Internet. The Web uses the Hypertext Transfer Protocol (HTTP protocol), as one of the languages spoken over the Internet, to transmit data. The Web also utilizes browsers to access Web documents called Web pages that are linked to each other via hyperlinks. Web documents also contain graphics, sounds, text and video.

The Web is just one of the ways that information can be disseminated over the Internet. The Internet, not the Web, is also used for e-mail, relies on Simple Mail Transfer Protocol (SMTP, the standard protocol used in the Internet for mail transfer and forwarding), Usenet, news groups, instant messaging and File Transfer Protocol (FTP, a network protocol used to transfer data from one computer to another through a network, such as the Internet). So the Web is just a portion of the Internet, albeit a large portion, but the two terms are not synonymous and should not be confused.

1.4.2  How the Internet came to be

Up until the mid-1960s, most computers
were stand-alone machines that did not connect to or communicate with other
computers. In 1962 J.C.R. Licklider, a professor at The Massachusetts Institute
of Technology (MIT, a private, coeducational research
university located in Cambridge, Massachusetts.), wrote a paper envisioning a
globally connected “Galactic Network” of computers [4]. The idea was far-out at
the time, but it caught the attention of Larry Roberts, a project manager at
the U.S. Defense Department’s Advanced Research Projects Agency (ARPA). In 1996
Roberts submitted a proposal to ARPA that would allow the agency’s numerous and
different computers to be connected in a network similar to Licklider’s Galactic
Network.

Robert’s proposal was accepted, and work
began on the “ARPANET”, which would in time become what we know as today’s
Internet. The first “node” on the ARPANET was installed at UCLA in 1969 and
gradually, throughout the 1970s, universities and defense contractors working
on ARPA projects began to connect to the ARPANET.

In 1973, the U.S. Defense Advanced
Research Projects Agency (DARPA) initiated another research program to allow
networked computers to communicate transparently across multiple linked
networks. Whereas the ARPANET was just one network, the new project was
designed to be a “network of networks”. According to Vint Cerf, widely regarded
as one of the “fathers” of the Internet, the Internetting project and the
system of networks which emerged from the research were known as the “Internet”
[9].

It wasn’t until the mid 1980s, with the
simultaneous explosion in the use of personal computers, and the widespread
adoption of a universal standard of Internet communication called Transmission
Control Protocol/Internet Protocol (TCP/IP), that the Internet became widely
available to anyone desiring to connect to it. Other government agencies
fostered the growth of the Internet by contributing communications “backbones”
that were specifically designed to carry Internet traffic. By the late 1980s,
the Internet had grown from its initial network of a few computers to a robust
communications network supported by government and commercial enterprises
around the world.

Despite this increased acceptability,
the Internet was still primarily a tool for academics environment and
government contractors well into the early 1990s. As more and more computers are
connected to the Internet, users began to demand tools that would allow them to
search for and locate text and other files on computers anywhere on the net.

1.4.3  Early Net Search Tools

In this section, we will trace the development of the early Internet
search tools, and show how their limitations ultimately spurred the popular
acceptance of the web. This historical background, while it is very fascinating
in its own right, lays the foundation for understanding why the Invisible Web
could arise in the first place.

Although sophisticated search and
information retrieval techniques dated back to the late 1950s and early ‘60s,
these techniques were used primarily in closed or proprietary systems. Early
Internet search and retrieval tools lacked even the most basic capabilities,
primarily because it was thought that traditional information retrieval techniques
would not work on an open, unstructured information universe like the Internet.

Accessing a file on the Internet was a
two-part process. First, you needed to establish direct connection to the
remote computer where the file was located using a terminal emulation program
called Telnet. Telnet is a terminal emulation program that runs on your
computer allowing you to access a remote computer via a TCP/IP network and
execute commands on that computer as if you were directly connected to it. Many
libraries offered telnet access to their catalogs. Then you needed to use
another program, called a File Transfer Protocol (FTP) client, to fetch the
file itself. File Transfer Protocol (FTP) is a set of rules for sending and
receiving files of all types between computers connected to the Internet. For
many years, to access a file, it was necessary to know both the address of the
computer and the exact location and name of the file you were looking for, that
is, there were no search engines or other file-finding tools like the ones we
are familiar with today.

Thus “search” often meant sending a
request for help to an e-mail message list or discussion forum and hoping some
kind soul would respond with the details you needed to fetch the file you were
looking for. The situation improved somewhat with the introduction of
“anonymous” FTP servers, which were centralized file-servers specifically
intended for enabling the easy sharing of files. The servers were anonymous
because they were not password protected, that is, anyone could simply log on
and request any file on the system.

Files on FTP servers were organized in
hierarchical directories, much like files are organized in hierarchical folders
on personal computer systems today. The hierarchical structure made it easy for
the FTP server to display a directory listing of all the files stored on the
server, but you still needed good knowledge of the contents of the FTP server.
If the file you were looking for didn’t exist on the FTP server you were logged
into, you were out of luck.

The first true search tool for files
stored on FTP servers was called Archie, created in 1990 by a small team of
systems administrators and graduate students at McGill University in Montreal. Archie
was the prototype of today’s search engines, but it was primitive and extremely
limited compared to what we have today. Archie roamed the Internet searching
for files available on anonymous FTP servers, downloading directories listings
of every anonymous FTP server it could find. These listings were stored in a
central, searchable database called the Internet Archives Database at McGill
University, and were updated monthly.

Although it represented a major step
forward, the Archie database was still extremely primitive, limiting searches
to a specific file name, or for computer programs that performed specific
functions. Nonetheless, it proved extremely popular because nearly 50% of
Internet traffic to Montreal in the early ‘90s was Archie related, according to
Deutsch [10], who headed up the McGill University Archie team.

“In the brief period following the
release of Archie, there was an explosion of Internet-based research projects,
including WWW, Gopher, WAIS, and others” [4]. Each explored a different area of
the Internet information problem space, and each offered its own insights into
how to build and deploy Internet-based services. The team licensed Archie to
others, with the first shadow sites launched in Australia and Finland in 1992.
The Archie network reached a peak of 63 installations around the world by 1995.

Gopher, an alternative to Archie, was
created by Mark McCahill and his team at the University of Minnesota in 1991
and was named for the university’s mascot, the Golden Gopher. Gopher
essentially combined the Telnet and FTP protocols, allowing users to click
hyperlinked menus to access information on demand without resorting to
additional commands. Using a series of menus that allowed the user to drill
down through successively more specific categories, users could ultimately
access the full text of documents, graphics, and even music files, though not
integrated in a single format. Gopher made it easy to browse for information on
the Internet.

According to Gopher creator McCahill,
“Before Gopher there wasn’t an easy way of having the sort of big distributed
system where there were seamless pointers between stuff on one machine and
another machine”. You had to know the name of this machine and if you wanted to
go over here you had to know its name.

“Gopher takes care of
all that stuff for you. So navigating around Gopher is easy. It points and
clicks typically. So it’s something that anybody could use to find things. It’s
also very easy to put information up so a lot of people started running servers
themselves and it was the first of the easy-to-use, no fuss, you can just crawl
around and look for information tools. It was the one that wasn’t written for
techies” [4].

Gopher’s “no muss, no fuss” interface
was an early precursor of what later evolved into popular Web directories like
Yahoo!. “Typically you set this up so that you can start out with a sort of
overview or general structure of a bunch of information, choose the items that
you’re interested in to move into a more specialized area and then either look
at items by browsing around and finding some documents or submitting searches”
[4].

A problem with Gopher was that it was designed to provide a listing of files available on computers in a specific location – the University of Minnesota, for example. While Gopher servers were searchable, there was no centralized directory for searching all other computers that were both using Gopher and connected to the Internet, or “Gopherspace” as it was called. In November 1992, Fred Barrie and Steven Foster of the University of Nevada System Computing Services group solved this problem, creating a program called Veronica, a centralized Archie-like search tool for Gopher files. In 1993 another program called Jughead added keyword search and Boolean operator capabilities to Gopher search. Keyword is a word or phrase entered in a query form that a search system attempts to match in text documents in its database. Boolean is a system of logical operators (AND, OR, NOT) that allows true-false operations to be performed on search queries, potentially narrowing or expanding results when used with keywords.

Related