Removing image partial blur is of great practical importance. However, as existing recovery techniques usually assume a one-layer clear image model, they can not characterize the actual generation process of partial blurs. In this paper, a two-layer image model is investigated. Based on the study of partial blur generation process, a novel recovery technique is proposed for a single input image. Both foreground and background layers are recovered simultaneously with the help of the matting technique, powerful image prior models, and user assistance. The effectiveness of the proposed approach is demonstrated by extensive experiments on image recovery and synthesis on real data.