—User profiles derived from Web navigation data are used in important e-commerce applications such as Web personalization, recommender systems, and Web analytics. In the open en...
Web search engines crawl the web to fetch the data that they index. In this paper we re-examine that need, and evaluate the network costs associated with data acquisition, and alt...
Nick Craswell, Francis Crimmins, David Hawking, Al...
The rapid growth of the World-Wide Web poses unprecedented scaling challenges for general-purpose crawlers and search engines. In this paper we describe a new hypertext resource d...
Soumen Chakrabarti, Martin van den Berg, Byron Dom
Search engines are useful because they allow the user to nd information of interest from the World-Wide Web. These engines use a crawler to gather information from Web sites. Howev...
Comprehensive coverage of the public web is crucial to web search engines. Search engines use crawlers to retrieve pages and then discover new ones by extracting the pages' o...