The multi-view video coding improves the coding efficiency by utilizing motion-compensated prediction (MCP) and disparity-compensated prediction (DCP). However, the complexity of the inter frame prediction is very high, especially when the rate-distortion optimization is used. This paper presents a fast inter frame prediction algorithm to reduce the complexity. Firstly the prediction type is decided according to reference frames. Then some unuseful search regions in view direction are removed. Finally a fast inter mode decision strategy is proposed based on the relationship between MCP and DCP. Experimental results verify that the proposed algorithm can greatly increase the speed of prediction with negligible loss of coding efficiency.