Towards a Content-Provider-Friendly Web Page Crawler

Search engine quality is impacted by two factors: the quality of the ranking/matching algorithm used and the freshness of the search engine’s index, which maintains a “snapshot” of the Web. Web crawlers capture web pages and refresh the index, but this is always a never-ending quest, as web pages get updated frequently (and thus have to be re-crawled). Knowing when to re-crawl a web page is fundamentally linked to the freshness of the index, given the size of the Web today and the inherent resource constraints: re-crawling too frequently leads to wasted bandwidth, recrawling too infrequently brings down the quality of the search engine. In this work, we address the scheduling problem for web crawlers, with the objective of optimizing the quality of the index (i.e., maximize the freshness probability of the local repository as well as of the index). Towards this, we utilize feedback from the users (content providers) on when their web pages are updated and consider the entire spe...

• Type: Conference
• Year: 2007
• Where: WEBDB
• Authors: Jie Xu, Qinglan Li, Huiming Qu, Alexandros Labrinidis
• Keywords: Internet Technology, Search Engine, Search Engine Index, Web Pages, WEBDB 2007