Automatic image annotation is a promising way to achieve more effective image management and retrieval by using keywords. However, system performances of the existing state-of-the-art keyword annotation schemes are often not so satisfactory. Therefore, image annotation refinement is crucial to improve the imprecise annotation results. In this paper, a novel approach is developed to automatically refine the initial annotation of images. First, for a query image, the candidate annotations are obtained by a step-up model-based algorithm using perceptual visual characteristic. Then, a refine algorithm, fast random walk with restart is used to re-rank the candidate annotations and the top ones are reserved as the final annotations. Experiments conducted on the typical Corel dataset shows that the proposed scheme can effectively improve the automatic annotation performance.