Sketching Information Divergences

When comparing discrete probability distributions, natural measures of similarity are not p distances but rather are informationdivergences such as Kullback-Leibler and Hellinger. This paper considers some of the issues related to constructing small-space sketches of distributions, a concept related to dimensionality-reduction, such that these measures can be approximately computed from the sketches. Related problems for p distances are reasonably well understood via a series of results including Johnson, Lindenstrauss [27, 18], Alon, Matias, Szegedy [1], Indyk [24], and Brinkman, Charikar [8]. In contrast, almost no analogous results are known to date about constructing sketches for the information-divergences used in statistics and learning theory.

• Type: Conference
• Year: 2007
• Where: COLT
• Authors: Sudipto Guha, Piotr Indyk, Andrew McGregor
• Keywords: COLT 2007, Discrete Probability Distributions, Machine Learning, Natural Measures, Small-space Sketches