In this paper, we present our eSur (Event detection system on SURveillance video) system, which is derived from TRECVID'09 surveillance tasks. Currently, eSur attempts to detect two categories of events: 1) single-actor events (i.e., PersonRuns and ElevatorNoEntry) irrespective of any interaction between individuals, and 2) pair-activity events (i.e., PeopleMeet, PeopleSplitUp, and Embrace) involves more than one individual. eSur consists of three major stages, i.e., preprocessing, event classification, and postprocessing. The preprocessing involves view classification, background subtraction, head-shoulder detection, human body detection and object tracking. Event classification fuses One-vs.-All SVM and rule-based classifiers to identify single-actor and pair-activity events in an ensemble way. To reduce false alarms, we introduce prior knowledge into the post-processing, and in particular, we apply a so-called event merging process over TRECVID dataset. Extensive experiments h...