Multicasting a Changing Repository

Web crawlers generate significant loads on Web servers, and are difficult to operate. Instead of repeatedly running crawlers at many "client" sites, we propose a central crawler and Web repository that multicasts appropriate subsets of the central repository, and their subsequent changes, to subscribing clients. Loads at Web servers are reduced because a single crawler visits the servers, as opposed to all the client crawlers. In this paper we model and evaluate such a central Web multicast facility for subscriber clients, and for mixes of subscriber and one-time downloader clients. We consider different performance metrics and multicast algorithms for such a multicast facility, and develop guidelines for its design under various conditions.

• Type: Conference
• Year: 2003
• Where: ICDE
• Authors: Wang Lam, Hector Garcia-Molina
• Keywords: Client Crawlers, Database, ICDE 2003, Web Crawlers, Web Multicast Facility