We propose a new system to mine visual knowledge on the Web. There are huge image data as well as text data on the Web. However, mining image data from the Web is paid less attention than mining text data, since treating semantics of images are much more difficult. In this paper, we propose introducing a latest image recognition technique, which is the bag-of-keypoints representation[1], into Web image-gathering task. By the experiments we show the proposed system outperforms our previous systems and Google Image search greatly. Categories and Subject Descriptors: I.4 [Image Processing and Computer Vision]: Miscellaneous General Terms: Algorithms, Experimentation, Measurement