This paper proposes a new image compression scheme by introducing visual patterns to nonlinear interpolative vector quantization (IVQ). Input images are first distorted by a generic down-sampling so that some details are removed before compression. Then, the distorted images are compressed lossly by traditional image coding scheme and transmitted to the decoder. In the decoder side, VQ indices are extracted from the decoded images to reproduce the removed details from a pre-trained codebook. One of main contributions in this paper is, we introduce visual patterns on designing the codebook, where only removed details that contain visual patterns and their original counterparts as pairs are trained. Experimental results show: (1) visual pattern blocks are easy to form clusters than original blocks; (2) the proposed scheme achieves much better performance over JPEG in terms of visual quality and PSNR.