Treating visual object tracking as foreground and background classification problem has attracted much attention in the past decade. Most methods adopt mean shift or brute force search to perform object tracking on the generated probability map, which is obtained from the classification results; however, performing probabilistic object tracking on the probability map is almost unexplored. This paper proposes a novel observation model which is suitable to perform this task. The observation model considers both region and boundary cues on the probability map, and can be computed very efficiently by using the integral image data structure. Extensive experiments are carried out on several challenging image sequences, which include abrupt motion change, background clutter, partial occlusion, and significant appearance change. Quantitative experiments are further performed with several related trackers on a public benchmark dataset. The experimental results demonstrate the effectiveness of ...