Distributed community crawling

The massive distribution of the crawling task can lead to inefficient exploration of the same portion of the Web. We propose a technique to guide crawlers exploration based on the notion of Web communities. The stability properties of the method can be used as an implicit coordination mechanism to increase the efficiency of the crawling task. Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval - clustering, search process, information filtering; H.5.4 [Information interfaces and presentation]: Hypertext/Hypermedia - navigation General Terms Algorithms, Experimentation Keywords Distributed Crawling, Web Metrics, Web Communities

• Type: Conference
• Year: 2004
• Where: WWW
• Authors: Fabrizio Costa, Paolo Frasconi
• Keywords: Crawling Task, Implicit Coordination Mechanism, Internet Technology, Web Communities, WWW 2004