In this paper a real-time anomaly detection system for video streams is proposed. Spatio-temporal features are exploited to capture scene dynamic statistics together with appearance. Anomaly detection is performed in a non-parametric fashion, evaluating directly local descriptor statistics. A method to update scene statistics, to cope with scene changes that typically happen in real world settings, is also provided. The proposed method is tested on publicly available datasets. Categories and Subject Descriptors H.3.1 [Information Systems]: Content Analysis and Indexing; H.5.1 [Multimedia Information Systems]: Video General Terms Algorithms, Experimentation Keywords Anomaly detection, surveillance, local descriptors, action recognition, spatio-temporal interest points