Information linkage is becoming more and more important in this digital age. In this paper, we propose a concept tracking method, which links the news stories with the same topic across multiple sources. The semantic linkage between the news stories is reflected in combination of both of their visual appearance and their spoken language content. Visually, each news story is represented by a set of key-frames with or without detected faces. The facial key-frames are linked based on the analysis of the extended facial regions, and the non-facial key-frames are correlated using the global Affine matching. The language similarity is expressed in terms of the normalized text similarity between the stories’ keywords. The output results of the story linking approach are further used in a story ranking task, which indicate the interesting level of the stories. The proposed semantic linking framework and the story ranking method have been tested on a set of 60 hours open-benchmark TRECVID v...