Incremental quantile estimation for massive tracking

Data--call records, internet packet headers, or other transaction records--are coming down a pipe at a ferocious rate, and we need to monitor statistics of the data. There is no reason to think that the data are normally distributed, so quantiles of the data are important to watch. The probe attached to the pipe has only limited memory, though, so it is impossible to compute the quantiles by sorting the data. The only possibility is to incrementally estimate the quantiles as the data fly by. This paper provides such an incremental quantile estimator. It resembles an exponentially weighted moving average in form, processing and memory requirements, but it is based on stochastic approximation so we call it an exponentially weighted stochastic approximation or EWSA. Simulations show that the EWSA outperforms other kinds of incremental estimates that also require minimal main memory, especially when extreme quantiles are tracked for patterns of behavior that change over time. Use of the E...

• Type: Conference
• Year: 2000
• Where: KDD
• Authors: Fei Chen, Diane Lambert, José C. Pinheiro
• Keywords: Data Mining, Exponentially Weighted Moving Average, KDD 2000, Quantiles, Stochastic Approximation