In this paper we present a new architecture for face recognition with a single reference image, which completely separates the training process from the recognition process. In the training stage, by using a database containing various individuals, the spatial relations between face components are represented by two Hidden Markov Models (HMMs), one modeling within-subject similarities, and the other modeling inter-subject differences. This allows us during the recognition stage to take a pair of face images, neither of which has been seen before, and to determine whether or not they come from the same individual. Whilst other face-recognition HMMs use Maximum Likelihood criterion, we test our approach using both Maximum Likelihood and Maximum a Posteriori (MAP) criterion, and find that MAP provides better results. Importantly, the training database can be entirely separated from the gallery and test images: this means that adding new individuals to the system can be done without re-tr...