In this paper, we propose a new video genre detection using semantic classification with multi-modal features. MPEG-7 audio-visual descriptors are used as multi-modal features. From the low-level multimodal features, genre as high-level semantic meaning is detected by using GINI index in Classification And Regression Tree (CART) algorithm. Experimental results show that the proposed method is useful to detect video genre automatically with a high detection rate.