The adaptive and powerful coding schemes in H.264 provide significant coding efficiency and some additional merits like error resilience and network friendliness. In spite of these outstanding features, it is not easy to implement H.264 codec as a real-time system due to its high requirement of memory bandwidth and intensive computation. Although the variable block size motion compensation using multiple reference frames is one of the key coding tools to bring about its main performance gain, it demands substantial computational complexity due to exhaustive search among all possible combinations of coding modes. Many existent fast motion estimation algorithms are not suitable for H.264 having variable motion block sizes. In this paper, we propose the motion field adaptive search using the hierarchical block structure based on the diamond search applicable to variable motion block sizes.