Sciweavers

WWW
2004
ACM
15 years 6 days ago
Design of a crawler with bounded bandwidth
This paper presents an algorithm to bound the bandwidth of a Web crawler. The crawler collects statistics on the transfer rate of each server to predict the expected bandwidth use...
Michelangelo Diligenti, Marco Maggini, Filippo Mar...