High order features have been proposed to incorporate geometrical information into the "bag of feature" representation. We propose algorithms to perform fast weakly supervised object categorization and localization with high order features. To this end, we first use Hough transform method to identify translation and scale invariant high order features co-occurring in two images. The co-occurrence is used to calculate a kernel for a SVM. Then, we propose an efficient algorithm for localization with high order features. A naive way would be to apply the SVM for all possible subwindows, which requires O(SM) kernel computations per image, where S is the number of support vectors, and M is the number of possible subwindows in an image. The algorithm collects the weights of high order features for the subwindows while calculating kernel value for the entire image, and thus reduces the kernel computations to O(S).