This paper presents a simple and effective image retargeting method that preserves visually important parts while reducing unwanted distortions of an image. Our approach is based on a novel importance diffusion scheme, which propagates importance of removed pixels to their neighbors for preserving visual contexts and avoiding over-shrinkage of unimportant parts. Importance diffusion enables even a simple row/column removal method, which removes the least important rows/columns repeatedly, to produce visually pleasant results. It also provides control over the trade-off between uniform and non-uniform sampling for the row/column removal and seam carving methods. Experimental result demonstrates that importance diffusion successfully improves the retargeting results of row/column removal and seam carving.