An Incremental Prefix Filtering Approach for the All Pairs Similarity Search Problem

Given a set of records, a threshold value t and a similarity function, we investigate the problem of finding all pairs of records such that similarity between each pair is above t. We propose several optimizations on the existing approaches to solve the problem. Our algorithm outperforms the state-of-the-art algorithms in the case with large and high-dimensional datasets. The speedup we achieved varied from 30% to 4-x depending on the similarity threshold and the dataset properties.

• Type: Conference
• Year: 2010
• Where: APWEB
• Authors: Hoang Thanh Lam, Dinh Viet Dung, Raffaele Perego, Fabrizio Silvestri
• Keywords: APWEB 2010, Internet Technology, Similarity Function, Similarity Threshold, Threshold Value