Variational Bayes Solution of Linear Neural Networks and Its Generalization Performance

It is well-known that, in unidentifiable models, the Bayes estimation provides much better generalization performance than the maximum likelihood (ML) estimation. However, its accurate approximation by Markov chain Monte Carlo methods requires huge computational costs. As an alternative, a tractable approximation method, called the variational Bayes (VB) approach, has recently been proposed and been attracting people’s attention. Its advantage over the expectation maximization (EM) algorithm, often used for realizing the ML estimation, has been experimentally shown in many applications, nevertheless, has not been theoretically shown yet. In this paper, through the analysis of the simplest unidentifiable models, we theoretically show some properties of the VB approach. We first prove that, in three-layer linear neural networks, the VB approach is asymptotically equivalent to a positive-part James-Stein type shrinkage estimation. Then, we theoretically clarify its free energy, gene...

• Type: Journal
• Year: 2007
• Where: NECO
• Authors: Shinichi Nakajima, Sumio Watanabe
• Keywords: Bayes Estimation, Estimation, NECO 2007, VB Approach