In this paper we propose a real-time method for tracking hands through image sequences. Our method combines efficiently calculated color likelihood maps with a state-ofthe-art interest point tracker. For estimating color likelihoods we apply an integral image based calculation of multivariate Gaussians which are compared by the KullbackLeibler distance. These likelihood maps are then passed to a modified version of the Maximally Stable Extremal Region (MSER)-tracker. The proposed algorithm allows to robustly track hands through image sequences and additionally provides accurate hand segmentations per frame. Experimental evaluation proves the high accuracy of the segmentation results and a first application for human-computer interaction is presented.