Deterministic Latent Variable Models and Their Pitfalls

We derive a number of well known deterministic latent variable models such as PCA, ICA, EPCA, NMF and PLSA as variational EM approximations with point posteriors. We show that the often practiced heuristic of "folding-in" can lead to overly optimistic estimates of the test-set log-likelihood and we verify this result experimentally. We trace this problem back to an infinitely negative entropy term that is ignored in the variational approximation.

• Type: Conference
• Year: 2008
• Where: SDM
• Authors: Max Welling, Chaitanya Chemudugunta, Nathan Sutter
• Keywords: Data Mining, Latent Variable Models, Negative Entropy Term, SDM 2008, Variational Em Approximations