Maximal Causes for Non-linear Component Extraction

We study a generative model in which hidden causes combine competitively to produce observations. Multiple active causes combine to determine the value of an observed variable through a max function, in the place where algorithms such as sparse coding, independent component analysis, or non-negative matrix factorization would use a sum. This max rule can represent a more realistic model of non-linear interaction between basic components in many settings, including acoustic and image data. While exact maximum-likelihood learning of the parameters of this model proves to be intractable, we show that efficient approximations to expectation-maximization (EM) can be found in the case of sparsely active hidden causes. One of these approximations can be formulated as a neural network model with a generalized softmax activation function and Hebbian learning. Thus, we show that learning in recent softmax-like neural networks may be interpreted as approximate maximization of a data likelihood. ...

• Type: Journal
• Year: 2008
• Where: JMLR
• Authors: Jörg Lücke, Maneesh Sahani
• Keywords: JMLR 2006, JMLR 2008, Model, Neural Networks, Non-Negative Matrix Factorization