In this paper a framework “Temporal-Vector Trajectory Learning” (TVTL) for human action recognition is proposed. In this framework, the major concept is that we would like to add the temporal information into the action recognition process. Base on this purpose, there are three kinds of temporal information, LTM, DTM, and TTM, being proposed. With the three kinds of proposed temporal information, the k-NN classifier based on the Mahanalobis distance metric do have better results than just using spatial information. The experimental results demonstrate that the method can recognize the actions well. Especially with our TTM and DTM framework, they do have great accuracy rates. Even with noisy data, the framework still have good performance.