The adaptive coding schemes in H.264 standard provide a significant coding efficiency and some additional features like error resilience and network friendliness. The variable block size motion compensation using multiple reference frames is one of the key H.264 coding elements to provide notable performance gain. However it is also the main culprit that increases the overall computational complexity. For this reason, this paper proposes a fast algorithm for variable block size motion estimation in H.264. In addition, we also propose a fast mode decision scheme by classifying modes based on rate-distortion cost. The experimental results show that the combined proposed methods provide significant improvement in processing speed without noticeable coding loss.