Due to rapid advances in video technology and biomedicine, there has been a tremendous growth in the volume of medical video data for recent years. Exploring the full potential of medical information from these data by semantic analysis is highly desirable and very useful. In this paper, we focus on how to classify images into semantic categories effectively and efficiently. There are two major contributions in this paper. The first contribution is that our proposed approach performs classification without segmentation and processing of individual objects. The second contribution is the proposed multiclass boosting algorithms that utilize the common features which can be shared among different semantic categories. Experimental results have demonstrated that our method is a promising strategy to solve the semantic classification problem for medical video data. To the best of our knowledge, no similar research has been reported in the biomedical image computing field and we expect our r...