This paper proposes a simple yet effective mechanism to select proper reference frames for H.264 motion estimation. Unlike traditional video codecs, H.264 permits more than one reference frame for increased precision in motion estimation. However, motion estimation is complicated by variable block-size motion estimation, which requires significant encoding complexity to identify the best inter-coding. Our smart selection mechanism selects suitable reference frames by means of a simple test, and only the selected frames will be searched further in the variable block size motion estimation. One major advantage of our mechanism is that it enables working with any existing motion search algorithms developed for the traditional single reference frame. Experimental results demonstrate the effectiveness of our proposed algorithm.