We propose a real-time pose invariant face recognition algorithm from a gallery of frontal images only. First, we modified the second order minimization method for active appearance model (AAM). This allows the AAM to have the ability of correct convergence with little loss of frame rate. Second, we proposed pose transforming matrix which can eliminate warping artifact of the warped face image from AAM fitting. This makes it possible to train a neural network as the face recognizer with one frontal face image of each person in the gallery set. Third, we propose a simple method for pose recognition by using neural networks to select proper pose transforming matrix. The proposed algorithm was evaluated on a set of 2000 facial images of 10 people (200 images for each person obtained at various poses) achieving a great improvement in recognition.