End-to-end distortion estimation plays an important role in error-resilient video coding. The intuitive method is to simulate the decoding process many times at the encoder, as used in the H.264 test model. However, the computing complexity is very high. In this paper, an optimum end-to-end distortion estimation scheme is proposed. Concretely, the correlations of the potentially propagated errors of the different frames are modeled based on the theoretical analysis, and then a block-based potential distortion tracking scheme with very low computing complexity is proposed. Statistics show that the gaps of the estimated distortions between the proposed algorithm and the H.264 test model become smaller and smaller when the times of simulated decoding in H.264 test model increases. In other words, the proposed algorithm is more accurate than H.264 test model. Moreover, an improved rate-distortion optimization algorithm based on the optimum end-to-end distortion estimation is proposed, wher...