The paper attempted the recognition of multiple drivers’ emotional state from physiological signals. The major challenge of the research is due to the severe inter-driver variation such that the features of different emotional state are high correlated, and it is found that simple decorrelation method cannot normalize the features well to achieve acceptable classification accuracy. Hence, in this paper, we propose to apply a latent variable to represent the hidden attribute of individual driver and use statistical training. In addition, we applied temporal constraints for the inference process to improve the recognition accuracy. Experimental results show that the proposed method outperform existing algorithms used for emotional state recognition.