Using Transitivity to Increase the Accuracy of Sample-Based Pearson Correlation Coefficients

Abstract. Pearson product-moment correlation coefficients are a wellpracticed quantification of linear dependence seen across many fields. When calculating a sample-based correlation coefficient, the accuracy of the estimation is dependent on the quality and quantity of the sample. Like all statistical models, these correlation coefficients can suffer from overfitting, which results in the representation of random error instead of an underlying trend. In this paper, we discuss how Pearson product-moment correlation coefficients can utilize information outside of the two items for which the correlation is being computed. By introducing a transitive relationship with one or more additional items that meet specified criterion, our Transitive Pearson product-moment correlation coefficient can significantly reduce the error, up to over 50%, of sparse, sample-based estimations. Finally, we demonstrate that if the data is too dense or too sparse, transitivity is detrimental in reducing the co...

• Type: Conference
• Year: 2010
• Where: DAWAK
• Authors: Taylor Phillips, Chris GauthierDickey, Ramki Thurimella
• Keywords: Correlation Coefficient, DAWAK 2010, Information Technology, Pearson Product-moment Correlation Coefficient, Sample-based Correlation