Improved robustness of signature-based near-replica detection via lexicon randomization

Detection of near duplicate documents is an important problem in many data mining and information filtering applications. When faced with massive quantities of data, traditional duplicate detection techniques relying on direct interdocument similarity computation (e.g., using the cosine measure) are often not feasible given the time and memory performance constraints. On the other hand, fingerprint-based methods, such as I-Match, are very attractive computationally but may be brittle with respect to small changes to document content. We focus on approaches to nearreplica detection that are based upon large-collection statistics and present a general technique of increasing their robustness via multiple lexicon randomization. In experiments with large web-page and spam-email datasets the proposed method is shown to consistently outperform traditional I-Match, with the relative improvement in duplicatedocument recall reaching as high as 40-60%. The large gains in detection accuracy are ...

• Type: Conference
• Year: 2004
• Where: KDD
• Authors: Aleksander Kolcz, Abdur Chowdhury, Joshua Alspector
• Keywords: Data Mining, Detection Accuracy, KDD 2004, Multiple Lexicon Randomization, Traditional Duplicate Detection