Abstract. A new algorithm for the incremental learning and non-intrusive tracking of the appearance of a previously non-seen face is presented. The computation is done in a causal fashion: the information for a given frame to be processed is combined only with the one of previous frames. To achieve this aim, a novel way for simultaneous and incremental computation of the Singular Value Decomposition (SVD) and the mean of the data is explained in this work. Previous developed methods about computing the SVD iteratively are taken into account and a novel way to extract the mean from a factorised matrix using SVD is obtained. Moreover, the results are achieved with linear computational cost and sublinear memory requirements with respect to the size of the data. Some experimental results are also reported.