Random Indexing using Statistical Weight Functions

Random Indexing is a vector space technique that provides an efficient and scalable approximation to distributional similarity problems. We present experiments showing Random Indexing to be poor at handling large volumes of data and evaluate the use of weighting functions for improving the performance of Random Indexing. We find that Random Index is robust for small data sets, but performance degrades because of the influence of high frequency attributes in large data sets. The use of appropriate weight functions improves this significantly.

• Type: Conference
• Year: 2006
• Where: EMNLP
• Authors: James Gorman, James R. Curran
• Keywords: Data Sets, Distributional Similarity Problems, EMNLP 2006, EMNLP 2007, Random Indexing