Video content is growing at an explosive rate nowadays. How to consume them efficiently is an important research point for years. Although the widely investigated video summarization solution can generate the main content of a video, it cannot ensure the coherence and apprehensibility of the original video. In this paper, we present a new framework called video shrinking to remove the video’s redundant information while keeping the integrality of the video content. Firstly, speech detection is performed to extract Candidate Deletion Shots (CDS), which have the property of low speech-ratio. Then, by combining the attention analysis and continuity analysis, CDS are refined to obtain the final temporal shrinking output. Subsequently, we further shrink the video spatially to adapt for the small screens of mobile devices. Experimental results demonstrate the effectiveness and efficiency of the proposed method.