Relative Loss Bounds for Multidimensional Regression Problems

We study on-line generalized linear regression with multidimensional outputs, i.e., neural networks with multiple output nodes but no hidden nodes. We allow at the final layer transfer functions such as the softmax function that need to consider the linear activations to all the output neurons. The weight vectors used to produce the linear activations are represented indirectly by maintaining separate parameter vectors. We get the weight vector by applying a particular parameterization function to the parameter vector. Updating the parameter vectors upon seeing new examples is done additively, as in the usual gradient descent update. However, by using a nonlinear parameterization function between the parameter vectors and the weight vectors, we can make the resulting update of the weight vector quite different from a true gradient descent update. To analyse such updates, we define a notion of a matching loss function and apply it both to the transfer function and to the parameterizat...

• Type: Conference
• Year: 1997
• Where: NIPS
• Authors: Jyrki Kivinen, Manfred K. Warmuth
• Keywords: NIPS 1997, NIPS 2007, Parameter Vectors, Parameterization Function, Weight Vector