This paper describes a camera-based observation system for football games that is used for the automatic analysis of football games and reasoning about multi-agent activity. The observation system runs on video streams produced by cameras set up for TV broadcasting. The observation system achieves reliability and accuracy through various mechanisms for adaptation, probabilistic estimation, and exploiting domain constraints. It represents motions compactly and segments them into classified ball actions. Categories and Subject Descriptors I.2.10 [Artificial Intelligence]: Vision and Scene Understanding-Video analysis General Terms Motion tracking, Analysis of intentional activity Keywords video analysis, state estimation, object tracking, motion interpretation