Image quality assessment (IQA) aims to provide computational models to measure the image quality in a perceptually consistent manner. In this paper, a novel feature based IQA model, namely Riesz-transform based Feature SIMilarity metric (RFSIM), is proposed based on the fact that the human vision system (HVS) perceives an image mainly according to its low-level features. The 1st order and 2nd -order Riesz transform coefficients of the image are taken as image features, while a feature mask is defined as the edge locations of the image. The similarity index between the reference and distorted images is measured by comparing the two feature maps at key locations marked by the feature mask. Extensive experiments on the comprehensive TID2008 database indicate that the proposed RFSIM metric is more consistent with the subjective evaluation than all the other competing methods evaluated.