The Motion-Compensated Temporal Filtering (MCTF) is an innovative prediction scheme for video coding and has become the core technology of the coming video coding standard, MPEG21 part 13 - Scalable Video Coding (SVC). This paper provides the system analysis of MCTF for VLSI implementation, which includes computational complexity, external memory access, external storage size, and coding delay. The one-level MCTF is analyzed first, and a modified double current frames scheme will be introduced to address the external memory access penalty that results from fractional-pel Motion Compensation (MC). Then the analysis is extended to multi-level MCTF, in which many important system issues will be explored. Finally, a real-life test case will be given to compare the system requirements of many different MCTF schemes and the prediction scheme of H.264/AVC.