Fast Curvature Matrix-Vector Products

The method of conjugate gradients provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore a number of ways to adopt ideas from conjugate gradient in the stochastic setting, using fast Hessian-vector products to obtain curvature information cheaply. In our benchmark experiments the resulting highly scalable algorithms converge about an order of magnitude faster than ordinary stochastic gradient descent.

• Type: Conference
• Year: 2001
• Where: ICANN
• Authors: Nicol N. Schraudolph
• Keywords: Conjugate Gradient, Gradient Descent, ICANN 2001, Neural Networks, Stochastic Gradient Descent