Video information processing has been one of great challenging areas in the database community since it needs huge amount of storage space and processing power. In this paper, we investigate the problem of clustering large video data sets that are collections of video clips as foundational work for the subsequent processing such as video retrieval. A video clip, a sequence of video frames, is represented by a multidimensional data sequence, which is partitioned into video segments considering temporal relationship among frames, and then similar segments of the clip are grouped into video clusters. We present the effective video segmentation and clustering algorithm which guarantees the clustering quality to such an extent that satisfies predefined conditions, and show its effectiveness via experiments on various video data sets.