In this paper, we propose a general framework for real time video data mining to be applied to the raw videos (traffic videos, surveillance videos, etc.). We investigate whether the existing techniques would be applicable to this type of videos. Then, we introduce new techniques which are essential to process them in real time. The first step of our frame work for mining raw video data is grouping input frames to a set of basic units which are relevant to the structure of the video. We call this unit as segment. This is one of the most important tasks since it is the step to construct the building blocks for video database and video data mining. The second step is characterizing each segment to cluster into similar groups, to discover unknown knowledge, and to detect interesting patterns. To do this, we extract some features (motion, object, colors, etc.) from each segment. In our framework, we focus on motion as a feature, and study how to compute and represent it for further process...