This paper presents a new method for tracking feature points on the textureless surface of a moving object. We employ a local image basis as a descriptor of each point for dealing with intensity variances due to the relative motion of the object to the light source. In particular, we propose to adaptively update the basis for enhancing its capability as the tracking proceeds. We show that the performance of the method further improves when it is coupled with Harris feature detector at a very large integration scale.