This paper presents a content-based approach for understanding handball videos. Tracked players are characterized by their 2D trajectories in the court plane. The trajectories and their interactions are used to model visual semantics, i.e., the observed activity phases. To this end, hierarchical parallel semi-Markov models (HPaSMMs) are computed in order to take into account the temporal causalities of object motions. Players motions are characterized using velocity informations while their interactions are described by the distances between trajectories. We have evaluated our method on real video sequences, and have favorably compared with another method,i.e., hierarchical parallel hidden Markov models (HPaHMMs). Categories and Subject Descriptors I.4.8 [Computing Methodologies]: Image Processing and Computer Vision—Scene Analysis; I.5.4 [Computing Methodologies]: Pattern Recognition—Applications Keywords Sports videos, Computer vision, Semi-Markov models, Motion analysis, Patter...