In this paper, a novel method for accurate subject tracking, by selecting only tracked subject boundary edges in a video stream with a changing background and moving camera, is proposed. This boundary edge selection is achieved in two steps: (1) removing background edges using edge motion, and from the output of the previous step, (2) selecting boundary edges using a normal direction derivative of the tracked contour. Accurate tracking is based on reduction of the effects of irrelevant edges, by only selecting boundary edge pixels. In order to remove background edges using edge motion, the tracked subject motion is computed and edge motions and edges having different motion directions from the subjects are removed. In selecting boundary edges using the normal contour direction, the image gradient values on every edge pixel are computed, and edge pixels with large gradient values are selected. Multi-level Canny edge maps are used to obtain proper details of a scene. Multi-level edge ma...