Sketching unaggregated data streams for subpopulation-size queries

IP packet streams consist of multiple interleaving IP flows. Statistical summaries of these streams, collected for different measurement periods, are used for characterization of traffic, billing, anomaly detection, inferring traffic demands, configuring packet filters and routing protocols, and more. While queries are posed over the set of flows, the summarization algorithm is applied to the stream of packets. Aggregation of traffic into flows before summarization requires storage of per-flow counters, which is often infeasible. Therefore, the summary has to be produced over the unaggregated stream. An important aggregate performed over a summary is to approximate the size of a subpopulation of flows that is specified a posteriori. For example, flows belonging to an application such as Web or DNS or flows that originate from a certain Autonomous System. We design efficient streaming algorithms that summarize unaggregated streams and provide corresponding unbiased estimators for subpo...

• Type: Conference
• Year: 2007
• Where: PODS
• Authors: Edith Cohen, Nick G. Duffield, Haim Kaplan, Carsten Lund, Mikkel Thorup
• Keywords: Database, Efficient Streaming Algorithms, IP Packet Streams, PODS 2007, Summarization Requires Storage