Architectural design and evaluation of an efficient Web-crawling system

This paper presents an architectural design and evaluation result of an efficient Web-crawling system. The design involves a fully distributed architecture, a URL allocating algorithm, and a method to assure system scalability and dynamic reconfigurability. Simulation experiment shows that load balance, scalability and efficiency can be achieved in the system. Currently this distributed Web-crawling subsystem has been successfully integrated with WebGather, a well-known Chinese and English Web search engine, aiming at collecting all the Web pages in China and keeping pace with the rapid growth of Chinese Web information. In addition, we believe that the design can also be useful in other context such as digital library, etc.

• Type: Journal
• Year: 2002
• Where: JSS
• Authors: Hongfei Yan, Jianyong Wang, Xiaoming Li, Lin Guo
• Keywords: Chinese Web Information, Distributed Web-crawling Subsystem, JSS 2002, Web Search Engine