Illumination changes cause object appearance to change drastically and many existing tracking algorithms lack the capability to handle this problem. The Earth Mover's Distance (EMD) is a similarity measure that is more robust against illumination changes. However, EMD is computationally expensive and we therefore propose the Differential EMD (DEMD) algorithm which computes the derivative of the EMD with respect to the object location so that the EMD does not need to be computed for every location in the tracking window. The fast differential formula is derived based on the sensitivity analysis of the simplex method as applied to the EMD formula. To further reduce the computation, signatures, i.e., variable-size descriptions of distributions, are employed as an object representation. The new algorithm models local background scenes as well as foreground objects to handle scale changes in a principled way. Extensive quantitative evaluation of the proposed algorithm has been carried...