DeepBot: a focused crawler for accessing hidden web content

The crawler engines of today cannot reach most of the information contained in the Web. A great amount of valuable information is "hidden" behind the query forms of online databases, and/or is dynamically generated by technologies such as Javascript. This portion of the web is usually known as the Deep Web or the Hidden Web. We have built DeepBot, a prototype of hidden-web focused crawler able to access such content. DeepBot receives a set of domain definitions as an input, each one describing a specific data-collecting task and automatically identifies and learns to execute queries on the forms relevant to them. In this paper we describe the techniques employed for building DeepBot and report the experimental results obtained when testing it with several real world data collection tasks. Categories and Subject Descriptors H.2.5 [Database Management]: Heterogeneous Databases. H.2.8 [Database Management]: Database Applications - Data mining. H.3.4 [Information Storage and Ret...

• Type: Conference
• Year: 2007
• Where: DEEC
• Authors: Manuel Álvarez, Juan Raposo, Alberto Pan, Fidel Cacheda, Fernando Bellas, Victor Carneiro
• Keywords: Database Management, DEEC 2007, Hidden Web, Hidden-web Focused Crawler, Information Management