Continuous sampling from distributed streams

A fundamental problem in data management is to draw and maintain a sample of a large data set, for approximate query answering, selectivity estimation, and query planning. With large, streaming data sets, this problem becomes particularly difficult when the data is shared across multiple distributed sites. The main challenge is to ensure that a sample is drawn uniformly across the union of the data while minimizing the communication needed to run the protocol on the evolving data. At the same time, it is also necessary to make the protocol lightweight, by keeping the space and time costs low for each participant. In this paper, we present communication-efficient protocols for continuously maintaining a sample (both with and without replacement) from k distributed streams. These apply to the case when we want a sample from the full streams, and to the sliding window cases of only the W most recent elements, or arrivals within the last w time units. We show that our protocols are opti...

• Type: Journal
• Year: 2012
• Where: JACM
• Authors: Graham Cormode, S. Muthukrishnan, Ke Yi, Qin Zhang
• Keywords: Approximate Query, Distributed And Parallel Computing, JACM 2012, Selectivity Estimation, Space Costs