Entropies and cross-entropies of exponential families

Statistical modeling of images plays a crucial role in modern image processing tasks like segmentation, object detection and restoration. Although Gaussian distributions are conveniently handled mathematically, the role of many other types of distributions has been revealed and emphasized by natural image statistics. In this paper, we consider a versatile class of distributions called exponential families that encompasses many well-known distributions, such as Gaussian, Poisson, multinomial, Gamma/Beta and Dirichlet distributions, just to name a few. For those families, we derive mathematical expressions for their Shannon entropy and cross-entropy, give a geometric interpretation, and show that they admit closedform formula up to some entropic normalizing constant depending on the carrier measure but independent of the member of the family. This allows one to design algorithms that can compare exactly entropies and cross-entropies of exponential family distributions although some of t...

• Type: Journal
• Year: 2010
• Where: ICIP
• Authors: Frank Nielsen, Richard Nock
• Keywords: Entropy, Exponential Families, ICIP 2010, Image, Image Processing