Detection of saliency regions in images is useful for object based image understanding and object localization. In our work, we investigate a saliency region detection algorithm based on the Human Visual Attention (HVA) model. In the first phase, we use mutual information and Probability-ofBoundary (PoB) for color saliency and edge detection respectively to filter SURF (Speeded Up Robust Features) key feature points found from the image. For the second phase, bipartite feature matching is deployed for further keypoint selection. We perform the two-phase keypoint filtering iteratively and give selected keypoints different weights for their importance. The final trimmed image is a rectangle region which approximates the distribution of remaining keypoints. We conduct our experiments on Corel Photo Library and MITCSAIL Objects and Scenes Database and demonstrate the effectiveness of our proposed algorithm.