This paper proposes a new framework to formulate the problem of rushes video summarization as an unsupervised learning problem. We pose the problem of video summarization as one of time-series clustering, and proposed Constrained Aligned Cluster Analysis (CACA). CACA combines kernel k-means, Dynamic Time Alignment Kernel (DTAK). Unlike previous work, that independently solve video segmentation and clustering, CACA jointly optimizes both and it is efficiently solved via dynamic programming. Experimental results on the TRECVID 2007 and 2008 BBC rushes video summarization databases validate the accuracy and effectiveness of CACA. Categories and Subject Descriptors I.2.10 [Artificial Intelligence]: Vision and Scene Understanding--Video analysis; H.3.1 [Information Storage and Retrieval]: Content Analysis and Indexing--Abstracting methods; H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems--Video General Terms Algorithms, Experimentation Keywords Constrained al...