Optimizing Frequency Queries for Data Mining Applications

Data mining algorithms use various Trie and bitmap-based representations to optimize the support (i.e., frequency) counting performance. In this paper, we compare the memory requirements and support counting performance of FP Tree, and Compressed Patricia Trie against several novel variants of vertical bit vectors. First, borrowing ideas from the VLDB domain, we compress vertical bit vectors using WAH encoding. Second, we evaluate the Gray code rankbased transaction reordering scheme, and show that in practice, simple lexicographic ordering, obtained by applying LSB Radix sort, outperforms this scheme. Led by these results, we propose HDO, a novel Hamming-distance-based greedy transaction reordering scheme, and aHDO, a linear-time approximation to HDO. We present results of experiments performed on 15 common datasets with varying degrees of sparseness, and show that HDOreordered, WAH encoded bit vectors can take as little as 5% of the uncompressed space, while aHDO achieves similar co...

• Type: Conference
• Year: 2007
• Where: ICDM
• Authors: Hassan H. Malik, John R. Kender
• Keywords: Bit Vectors, Data Mining, ICDM 2007, Transaction Reordering Scheme, Vertical Bit Vectors