This paper introduces a clever way of computing inner products between images in order to drastically reduce the computational complexity of fitting appearance models to images. This speedup is possible since computing the hessian matrix for the parameter updates becomes several orders of magnitude faster which in turn has enormous impact on applications. Contrary to previous work within the area, our method has no negative side-effects on accuracy, convergence and robustness of the algorithm to which it is applied. The inner products are computed in a linear space defined by the appearance model which makes the computation time independent of the model image size.