Identifying redundant search engines in a very large scale metasearch engine context

For a given set of search engines, a search engine is redundant if its searchable contents can be found from other search engines in this set. In this paper, we propose a method to identify redundant search engines in a very large-scale metasearch engine context. The general problem is equivalent to an NP hard problem – the set-covering problem. Due to the large number of search engines that need to be considered and the large sizes of these search engines, approximate solutions must be developed. In this paper, we propose a general methodology to tackle this problem and within the context of this methodology, we propose several new heuristic algorithms for solving the set-covering problem. Categories and Subject Descriptors H.3.4 [Information Storage and Retrieval]: Systems and Software – Distributed Systems; H.3.5: Online Information Services –Web-based Services. General Terms Algorithms, Performance, Design, Experimentation. Keywords Redundant search engine identification, se...

• Type: Conference
• Year: 2006
• Where: WIDM
• Authors: Ronak Desai, Qi Yang, Zonghuan Wu, Weiyi Meng, Clement T. Yu
• Keywords: Engine, Redundant Search Engine, Search Engine, WIDM 2006