Convolutional learning of spatio-temporal features

Abstract. We address the problem of learning good features for understanding video data. We introduce a model that learns latent representations of image sequences from pairs of successive images. The convolutional architecture of our model allows it to scale to realistic image sizes whilst using a compact parametrization. In experiments on the NORB dataset, we show our model extracts latent “flow fields” which correspond to the transformation between the pair of input frames. We also use our model to extract low-level motion features in a multi-stage architecture for action recognition, demonstrating competitive performance on both the KTH and Hollywood2 datasets. Key words: unsupervised learning, restricted Boltzmann machines, convolutional nets, optical flow, video analysis, activity recognition

• Type: Conference
• Year: 2010
• Where: ECCV
• Keywords: Computer Vision, Convolutional Architecture, ECCV 2010, Latent Representations, Model