In this work, we investigate how to propagate annotated labels for a given single image from the image-level to their corresponding semantic regions, namely Label-toRegion (L2R), by utilizing the auxiliary knowledge from Internet image search with the annotated image labels as queries. A nonparametric solution is proposed to perform L2R for single image with complete labels. First, each label of the image is used as query for online image search engines to obtain a set of semantically related and visually similar images, which along with the input image are encoded as Bags-of-Hierarchical-Patches. Then, an efficient two-stage feature mining procedure is presented to discover those input-image specific, salient and descriptive features for each label from the proposed Interpolation SIFT (iSIFT) feature pool. These features consequently constitute a patch-level representation, and the continuitybiased sparse coding is proposed to select few patches from the online images with preferen...