Given a pair of images represented using bag-of-visual words and a label corresponding to whether the images are “related”(must-link constraint) or “unrelated” (must not link constraint), we address the problem of selecting a subset of visual words that are salient to the relevance between the image pair. An efficient online feature selection algorithm is presented based on the dual-gradient descent approach. Side information in the form of pair-wise constraints is incorporated into the feature selection stage, providing the user with flexibility to use unsupervised or semi-supervised
algorithm at a later stage. Correlated subsets of visual words, usually resulting from hierarchical quantization process (called groups), are exploited to select a significantly reduce the vocabulary size. A subset of features is selected such that the distance computed using these features satisfies the given pairwise constraints. A group-LASSO regularizer is used to drive as many feature weight...