How to scalably and accurately skip past streams

Data stream methods look at each new item of the stream, perform a small number of operations while keeping a small amount of memory, and still perform muchneeded analyses. However, in many situations, the update speed per item is extremely critical and not every item can be extensively examined. In practice, this has been addressed by only examining every Nth item from the input; decreasing the input rate by a fraction 1/N, but resulting in loss of guarantees on the accuracy of the post-hoc analyses. In this paper, we present a technique of skipping past streams and looking at only a fraction of the input. Unlike traditional methods, our skipping is performed in a principled manner based on the “norm” of the stream seen. Using this technique on top of well-known sketches, we show several-fold improvement in the update time for processing streams with a given guaranteed accuracy, for a number of stream processing problems including data summarization, heavy hitters detection and s...

• Type: Conference
• Year: 2007
• Where: ICDE
• Authors: Supratik Bhattacharyya, André Madeira, S. Muthukrishnan, Tao Ye
• Keywords: Data Stream Methods, Database, ICDE 2007, Stream, Stream Processing