We consider how to exploit the correlation in image for compression by virtue of studying image patches in a nonparametric manner. Instead of extracting and recording parameters, our approach directly operates on image patches. The basic assumption is that a subset of image patches can be well inferred from the others; therefore, they can be removed at encoder only to be restored at decoder. Meanwhile, assistant information is transmitted for the restoration, which actually encodes the similarity between removed and preserved patches. The entire scheme is built upon an optimization framework, which is decoupled and solved accordingly.