Second Order Approximations for Probability Models

In this paper, we derive a second order mean field theory for directed graphical probability models. By using an information theoretic argument it is shown how this can be done in the absense of a partition function. This method is a direct generalisation of the well-known TAP approximation for Boltzmann Machines. In a numerical example, it is shown that the method greatly improves the first order mean field approximation. For a restricted class of graphical models, so-called single overlap graphs, the second order method has comparable complexity to the first order method. For sigmoid belief networks, the method is shown to be particularly fast and effective.

• Type: Conference
• Year: 2000
• Where: NIPS
• Authors: Hilbert J. Kappen, Wim Wiegerinck
• Keywords: Mean Field Theory, NIPS 2000, NIPS 2007, Order Mean Field, Order Method