Existing visual tracking methods are challenged by object and background appearance variations, which often occur in a long duration tracking. In this paper, we propose a combined feature evaluation approach in filter frameworks for adaptive object tracking. First, a feature set is constructed by combining color histogram (HC) and gradient orientation histogram (HOG), which gives a representation of both color and contour. Then, to adapt to the appearance changes of the object and its background, these features are assigned with different confidences adaptively to make the features with higher discriminative ability play more important roles in the instantaneous tracking. To keep the temporal consistency, the feature confidences are evaluated based on Kalman and Particle filters. Experiments and comparisons demonstrate that object tracking with evaluated features have good performance even when objects go across complex backgrounds.