Image quality assessment (IQA) is of great importance for many image processing applications. Some IQA indexes proposed recently more or less try to boost their performance to accord with human subjective evaluation by simulating human visual system (HVS). However, they do not take global salient features into consideration, because of the lack of methods with low computational complexity for simulating visual attention mechanism. This paper proposes a simpler and faster method to extract a saliency map from the reference image, and inserts saliency factors into existing IQA indexes. Experimental results for a set of intuitive examples as well as validation from a database of 982 images with different distortion types show that our improved IQA indexes are much closer to human opinion.