WIC: A General-Purpose Algorithm for Monitoring Web Information Sources

The Web is becoming a universal information dissemination medium, due to a number of factors including its support for content dynamicity. A growing number of Web information providers post near real-time updates in domains such as auctions, stock markets, bulletin boards, news, weather, roadway conditions, sports scores, etc. External parties often wish to capture this information for a wide variety of purposes ranging from online data mining to automated synthesis of information from multiple sources. There has been a great deal of work on the design of systems that can process streams of data from Web sources, but little attention has been paid to how to produce these data streams, given that Web pages generally require “pull-based” access. In this paper we introduce a new generalpurpose algorithm for monitoring Web information sources, effectively converting pull-based sources into push-based ones. Our algorithm can be used in conjunction with continuous query systems that ass...

• Type: Conference
• Year: 2004
• Where: VLDB
• Authors: Sandeep Pandey, Kedar Dhamdhere, Christopher Olston
• Keywords: Database, VLDB 2004, Web Information, Web Information Providers, Web Monitoring