Applying computer vision technology to IR (Infra-Red) images for UAV (Unmanned Aerial Vehicle) applications is difficult due to its characteristics which differ from common image processing. By combining visual categorization with low level IR image processing, this paper presents a framework for automatic labeling of IR images in probabilistic manner. We extract the features which contain temperature, texture and orientation information from the IR image, model visual categories by the distribution of features in terms of an extended visual vocabulary, and categorize IR image segments probabilistically. The proposed framework is demonstrated in experiments with high labeling accuracy, for near IR images of urban terrain taken from 100 feet altitude.