In this paper, we propose a hierarchical classifier structure for gender classification based on facial images by reducing the complexity of the original problem. In the proposed framework, we first train a classifier, which will properly divide the input images into several groups. For each group, we train a gender classifier, which is called expert. These experts can be any commonly used classifiers, such as Support Vector Machine (SVM) and neural network. The symmetrical characteristic of human face is utilized to further reduce the complexity. Moreover, we adopt soft assignment instead of hard one when dividing the input data, which can reduce the error introduced by the division. Experimental results demonstrate that our framework significantly improves the performance. Key words: Hierarchical classifiers, Gender classification, Multi-view facial images