Brute force and indexed approaches to pairwise document similarity comparisons with MapReduce

This paper explores the problem of computing pairwise similarity on document collections, focusing on the application of “more like this” queries in the life sciences domain. Three MapReduce algorithms are introduced: one based on brute force, a second where the problem is treated as large-scale ad hoc retrieval, and a third based on the Cartesian product of postings lists. Each algorithm supports one or more approximations that trade effectiveness for efficiency, the characteristics of which are studied experimentally. Results show that the brute force algorithm is the most efficient of the three when exact similarity is desired. However, the other two algorithms support approximations that yield large efficiency gains without significant loss of effectiveness. Categories and Subject Descriptors: H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval General Terms: Algorithms, Performance

• Type: Conference
• Year: 2009
• Where: SIGIR
• Authors: Jimmy J. Lin
• Keywords: Ad Hoc Retrieval, Algorithms, Brute Force, Information Retrieval, SIGIR 2009