Distributed monitoring of conditional entropy for anomaly detection in streams

In this work we consider the problem of monitoring information streams for anomalies in a scalable and efficient manner. We study the problem in the context of network streams where the problem has received significant attention. Monitoring the empirical Shannon entropy of a feature in a network packet stream has previously been shown to be useful in detecting anomalies in the network traffic. Entropy is an information-theoretic statistic that measures the variability of the feature under consideration. Anomalous activity in network traffic can be captured by detecting changes in this variability. There are several challenges, however, in monitoring this statistic. Computing the statistic efficiently is non-trivial. Further, when monitoring multiple features, the streaming algorithms proposed previously would likely fail to keep up with the everincreasing channel bandwidth of network traffic streams. There is also the concern that an adversary could attempt to mask the effect of his at...

• Type: Journal
• Year: 2010
• Where: IPPS
• Authors: Chrisil Arackaparambil, Sergey Bratus, Joshua Brody, Anna Shubina
• Keywords: Distributed And Parallel Computing, Entropy, IPPS 2010, Network Traffic, Statistic