Exploiting a support-based upper bound of Pearson's correlation coefficient for efficiently identifying strongly correlated pair

Given a user-specified minimum correlation threshold and a market basket database with N items and T transactions, an all-strong-pairs correlation query finds all item pairs with correlations above the threshold . However, when the number of items and transactions are large, the computation cost of this query can be very high. In this paper, we identify an upper bound of Pearson's correlation coefficient for binary variables. This upper bound is not only much cheaper to compute than Pearson's correlation coefficient but also exhibits a special monotone property which allows pruning of many item pairs even without computing their upper bounds. A Two-step All-strong-Pairs corrElation queRy (TAPER) algorithm is proposed to exploit these properties in a filter-and-refine manner. Furthermore, we provide an algebraic cost model which shows that the computation savings from pruning is independent or improves when the number of items is increased in data sets with common Zipf or li...

• Type: Conference
• Year: 2004
• Where: KDD
• Authors: Hui Xiong, Shashi Shekhar, Pang-Ning Tan, Vipin Kumar
• Keywords: All-strong-pairs Correlation Query, Data Mining, KDD 2004, Minimum Correlation Threshold, Two-step All-strong-Pairs CorrElation