A key element for video indexing and summarisation is the description of isolated events and actions. In the context of many sports the motion of the ball plays an essential role in describing events. Due to the difficulty of ball tracking, specially in standard broadcast video, this cue has been overlooked by most researchers, in particular for games of tennis, in which the ball resolution is very small and it moves very fast. A data association method has reported a high level of success on tennis ball tracking, but so far this tracker's output has only been processed by a method based on manually crafted rules for event recognition. This set of rules use cues such as proximity between ball and players or court lines. We present an HMM paradigm to automatically learn to identify events from ball trajectories and demonstrate that its ability to capture the dynamics of the ball movement lead to a much higher performance.