Object tracking based on Mean Shift (MS) algorithm has been very successful and thus receives significant research interests. Unfortunately, traditional MS based tracking only utilizes the gradient of the similarity function (SF), neglecting completely higher-order information of SF. The paper regards MS based tracking as an optimization problem, and proposes to make use of both the Gradient and Hessian of SF. Specifically, we introduce Newton algorithm with constant, unit step and Newton with varying step lengths, and Trust region algorithm. The advantage of exploiting higher-order information is that higher convergence rate and better performance are achieved. Diverse experiments are made to compare traditional MS based tracking with the proposed algorithms, showing that the proposed algorithms have better performance at comparable computational cost.