H.264/scalable video coding (SVC) is a new compression technique that can adapt to various network environments and applications. However, despite its outstanding performance, H.264/SVC has considerable complexity burden on decoding, especially in inverse transform and interpolation for motion compensation. We first analyze the Joint Scalable Video Model (JSVM) decoder to identify time-consuming modules of H.264/SVC. Based on the analysis, we present the single instruction multiple data (SIMD) Optimization methods for 4×4 inverse transform and interpolation for motion compensation with an efficient data structure. On a 2.4 GHz Intel Pentium IV processor, the proposed methods reduce considerably the computation time for inverse transform and interpolation for motion compensation, compared to JSVM software.