Detecting small targets is essential for mitigating the seabased Infrared search and track (IRST) problem. It is easy to detect small targets in homogeneous backgrounds such as the sky. When targets are on the border line of heterogeneous backgrounds such as the horizon in the sky and sea surface, solving the problem of detection becomes difficult. This paper presents a novel spatial filtering method, called Double Layered-Background Removal Filter (DL-BRF), for achieving high detection rates and low false alarm rates. DL-BRF consists of a Modified-Mean Subtraction Filter (M-MSF) and a consecutive Local-Directional Background Removal Filter (L-DBRF). M-MSF enhances the target signal and reduces background noise. L-DBRF removes horizontal structures, which upgrade the signal-to-clutter ratio and background suppression factor. L-DBRF used after M-MSF enhances the synergistic performance of horizontal target detection. We validate the superior performance of the proposed method via re...