This paper deals with motion estimation and segmentation in video sequences. Some methods of motion computation between two consecutive frames of a video sequence are based on the minimization of the square error of the prediction error. More robust estimators such as absolute value or M-estimators were proposed but these estimators loose their efficiency when the data do not have parametric distributions. We relax the parametric assumption on the prediction error distribution and propose to use a nonparametric estimator for the motion estimation : the entropy of the prediction error. We use the same criterion to perform a spatio-temporal segmentation of the sequence using an active contour algorithm. Segmentation and tracking tests on a textured synthetic and a real sequence, compared to a standard method in motion segmentation, tends to show that our method is more stable and accurate.