With the explosion of video data, video processing technologies have advanced quickly and been applied into many fields, such as advertisements, medical etc.. To fast search these video data, an important issue is to effectively organize videos by data compacting and indexing. However, practically, many useful distances for video comparison are suitable to human perception, but non-metric. Therefore, traditional high dimensional data structures can not be utilized to index videos directly when non-metric measures are applied. In this paper, we propose a compact video representation based on global summarization, by which each video in database is mapped into a digital string(a series of cluster id). Consequently, the inter-frame similarity measure is transformed into inter-cluster comparison. Then, we propose an efficient index strategy based on sequence decomposition and reconstruction, by which the spatial index methods can be utilized with non-metric measures for video similarity s...