— This paper makes new contributions in motion detection, object segmentation and trajectory estimation to create a successful object tracking system. A new efficient motion detection algorithm referred to as the flux tensor is used to detect moving objects in infrared video without requiring background modeling or contour extraction. The flux tensor-based motion detector when applied to infrared video is more accurate than thresholding ”hot-spots”, and is insensitive to shadows as well as illumination changes in the visible channel. In real world monitoring tasks fusing scene information from multiple sensors and sources is a useful core mechanism to deal with complex scenes, lighting conditions and environmental variables. The object segmentation algorithm uses level set-based geodesic active contour evolution that incorporates the fusion of visible color and infrared edge informations in a novel manner. Touching or overlapping objects are further refined during the segment...