Abstract— This paper describes a novel approach for incremental learning of human motion pattern primitives through on-line observation of human motion. The observed motion time series data stream is first stochastically segmented into potential motion primitive segments, based on the assumption that data belonging to the same motion primitive will have the same underlying distribution. The motion segments are tracted into a stochastic model representation, and automatically clustered and organized. As new motion patterns are observed, they are incrementally grouped together based on their relative distance in the model space. The resulting representation of the knowledge domain is a tree structure, with specialized motions at the tree leaves, and generalized motions closer to the root. The tree leaves, which represent the most specialized learned motion primitives, are then passed back to the segmentation algorithm, so that as the number of known motion primitives increases, the ac...