On derivation of stagewise second-order backpropagation by invariant imbedding for multi-stage neural-network learning

— We present a simple, intuitive argument based on “invariant imbedding” in the spirit of dynamic programming to derive a stagewise second-order backpropagation (BP) algorithm. The method evaluates the Hessian matrix of a general objective function efficiently by exploiting the multistage structure embedded in a given neural-network model such as a multilayer perceptron (MLP). In consequence, for instance, our stagewise BP can compute the full Hessian matrix “faster” than the standard method that evaluates the GaussNewton Hessian matrix alone by rank updates in nonlinear least squares learning. Through our derivation, we also show how the procedure serves to develop advanced learning algorithms;

• Type: Conference
• Year: 2006
• Where: IJCNN
• Authors: Eiji Mizutani, Stuart Dreyfus
• Keywords: General Objective Function, Hessian Matrix, IJCNN 2006, Stagewise Second-order Backpropagation