Distributed location aware web crawling

Distributed crawling has shown that it can overcome important limitations of the today's crawling paradigm. However, the optimal benefits of this approach are usually limited to the sites hosting the crawler. In this work, we propose a location-aware method, called IPMicra, that utilizes an IP address hierarchy, and allows crawling of links in a near optimal location aware manner. Categories and Subject Descriptors H.3.4 [Systems and Software]: Distributed Systems General Terms Performance Keywords location aware web crawling, distributed web crawling

• Type: Conference
• Year: 2004
• Where: WWW
• Authors: Odysseas Papapetrou, George Samaras
• Keywords: Internet Technology, IP Address Hierarchy, Location Aware Manner, Location Aware Web, WWW 2004