As one of the important research areas of multimodal interaction, sign language recognition (SLR) has attracted increasing interest. In SLR, especially on medium or large vocabulary, it is usually difficult or impractical to collect enough training data. Thus, how to improve the recognition on the limited training samples is a significant issue. In this paper, a simple but effective hierarchical voting classification (HVC) scheme for improving visual SLR, which makes efficient use of limited training data, is proposed. The key idea of HVC scheme is similar to but not the same as Bagging technique. Firstly, it constructs several training sets from the original training set in a combinatorial fashion to generate the corresponding continuous hidden Markov models (CHMM) ensemble. Then, it determines the ensemble output by appropriate local voting strategy. Finally, it obtains the final recognition result by the global voting. Experimental results show that the HVC scheme outperforms the c...