In this paper, we proposed an improved “bottom-up” variable-size block matching method. Different from previous work, the proposed method does not need any threshold during the matching, and we just keep all the motion vectors leading to the minimum matching error. A Marco-block mode prediction method is put forward to speed up the motion estimation procedure without introducing any loss to the prediction precision. The improved variable-size block matching algorithm can achieve exactly the same prediction precision as full-search based fixed-size block matching algorithm. In order to reduce the effect of illumination change on mode selection, we proposed an illumination removal method, which acts as a post-processing step to prevent the macro-blocks from over-splitting. Experiments show its encouraging performance.