The recognition of activities from sensory data is important in advanced surveillance systems to enable prediction of high-level goals and intentions of the target under surveillance. The problem is complicated by sensory noise and complex activity spanning large spatial and temporal extents. This paper presents a system for recognising high-level human activities from multi-camera video data in complex spaironments. The Abstract Hidden Markov mEmory Model (AHMEM) is used to deal with noise and scalability. M is an extension of the Abstract Hidden Markov Model (AHMM) that allows us to represent a richer class of both state-dependent and context-free behaviours. The model also supports integration with low-level sensory models and efficient probabilistic inference. We present experimental results showing the ability of the system to perform real-time monitoring and recognition of complex behaviours of people from observing their trajectories within a real, complex indoor environment.