The average-case complexity of counting distinct elements

We continue the study of approximating the number of distinct elements in a data stream of length n to within a (1? ) factor. It is known that if the stream may consist of arbitrary data arriving in an arbitrary order, then any 1-pass algorithm requires (1/ 2 ) bits of space to perform this task. To try to bypass this lower bound, the problem was recently studied in a model in which the stream may consist of arbitrary data, but it arrives to the algorithm in a random order. However, even in this model an (1/ 2 ) lower bound was established. This is because the adversary can still choose the data arbitrarily. This leaves open the possibility that the problem is only hard under a pathological choice of data, which would be of little practical relevance. We study the average-case complexity of this problem under certain distributions. Namely, we study the case when each successive stream item is drawn independently and uniformly at random from an unknown subset of d items for an unknown ...

• Type: Conference
• Year: 2009
• Where: ICDT
• Authors: David P. Woodruff
• Keywords: Database, ICDT 2009, Lower Bound, Lower Bound Subsumes, Random Uncorrelated Data