Hua et al. have proposed a stable and efficient tracking algorithm called “K-means tracker”[2, 3, 5]. This paper describes an adaptive non-target cluster center selection method that replaces the one used in K-means tracker where non-target cluster center are selected at fixed interval. Non-target cluster centers are selected from the ellipse that defines the area for searching the target object in K-means tracker by checking whether they have significant effects for the pixel classification and are dissimilar to any of the already-selected nontarget cluster centers. This ensures that all important non-target cluster centers will be picked up while avoiding selecting redundant non-target clusters. Through comparative experiments of object tracking, we confirmed that both the robustness and the processing speed could be improved with our method.