A self-adaptive Hidden Markov Model (SA-HMM) based framework is proposed for behavior recognition in this paper. In this model, if an unknown sequence cannot be classified into any trained HMMs, a new HMM will be generated and trained, where online training is applied on SA-HMMs to dynamically generate the high-level description of behaviors. The SA-HMMs based framework consists of training and classification stages. During the training stage, the state transition and output probabilities of HMMs can be optimized through the Gaussian Mixture Models (GMMs) so the generated symbols can match the observed image features within a specific behavior class. On the classification stage, the probability with which a particular HMM generates the test symbol sequence will be calculated, which is proportional to the likelihood.