Most affinity-based grouping methods only model the inclusive relation among the data. When the data set contains a significant amount of noise data that should not be included in any clusters, these methods are likely to lead to undesired results. To address this issue, this paper presents a new approach called bipolar grouping that is targeted on extracting the groups from the data while excluding the noise. This new approach incorporates both inclusive and exclusive relations among data, and a fixed-point procedure is proposed to find the stable groups. Its effectiveness and general applicability are demonstrated in two applications, including discovering common objects in images and tracking targets in clutter. Keywords-- Bipolar grouping, Common pattern discovery, Visual object tracking