This paper presents a novel non-iterative method to restore the out-of-focus part of an image. The proposed method first applies a robust local blur estimation to obtain a blur map of the image. The estimation uses the maximum of difference ratio between the original image and its two digitally re-blurred versions to estimate the local blur radius. Then adaptive least mean square filters based on the local blur radius and the image structure are applied to restore the image and to eliminate the sensor noise. Experimental results have shown that despite its low complexity the proposed method has a good performance at reducing spatially varying blur.