Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

We consider the task of estimating, from observed data, a probabilistic model that is parameterized by a finite number of parameters. In particular, we are considering the situation where the model probability density function is unnormalized. That is, the model is only specified up to the partition function. The partition function normalizes a model so that it integrates to one for any choice of the parameters. However, it is often impossible to obtain it in closed form. Gibbs distributions, Markov and multi-layer networks are examples of models where analytical normalization is often impossible. Maximum likelihood estimation can then not be used without resorting to numerical approximations which are often computationally expensive. We propose here a new objective function for the estimation of both normalized and unnormalized models. The basic idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise. With th...

• Type: Journal
• Year: 2012
• Where: JMLR
• Authors: Michael Gutmann, Aapo Hyvärinen
• Keywords: JMLR 2012, Maximum Likelihood Estimation, Natural Image Statistics, Probability Density Function, Programming Languages