In this paper we present an automatic facial expression recognition system that utilizes a semantic-based learning algorithm using the analytical hierarchy process (AHP). Although the effectiveness of low-level features in automatic facial expression recognition systems has been widely studied, the success is shadowed by the innate discrepancy between the machine and human perception to the image. The gap between low-level visual features and high-level semantics should be bridged in a proper way in order to construct a seamless automatic facial expression system satisfying the user perception. For this purpose, we use the AHP to provide a systematical way to evaluate the fitness of a semantic description used to interpret the emotion of a face image. A semantic-based learning algorithm is also proposed to choose the weights of low-level visual features. The weights are chosen such that the discrepancy between the facial expression recognition results obtained in terms of low-level fe...