Max-Margin Min-Entropy Models

We propose a new family of latent variable models called max-margin min-entropy (m3e) models, which define a distribution over the output and the hidden variables conditioned on the input. Given an input, an m3e model predicts the output with the smallest corresponding R´enyi entropy of generalized distribution. This is equivalent to minimizing a score that consists of two terms: (i) the negative log-likelihood of the output, ensuring that the output has a high probability; and (ii) a measure of uncertainty over the distribution of the hidden variables conditioned on the input and the output, ensuring that there is little confusion in the values of the hidden variables. Given a training dataset, the parameters of an m3e model are learned by maximizing the margin between the R´enyi entropies of the ground-truth output and all other incorrect outputs. Training an m3e can be viewed as minimizing an upper bound on a user-defined loss, and includes, as a special case, the latent suppor...

• Type: Journal
• Year: 2012
• Where: JMLR
• Authors: Kevin Miller, M. Pawan Kumar, Benjamin Packer, Danny Goodman, Daphne Koller
• Keywords: Hidden Variables, JMLR 2012, Latent Variable Models, Programming Languages, Support Vector Machine