A new down-conversion scheme in the DCT domain is presented, which can be used in decoders of DCT-compressed images and videos. The down-conversion in the transform domain generally requires lower computational complexity than the spatial domain down-conversion. The proposed method also requires computational complexity comparable to other DCT domain techniques, and it is optimal in MSE sense, whereas the others are not. We minimized the number of arithmetic operations without discarding any data in DCT coefficients. We first combined the DQ, IDCT, and spatial domain averaging together. To minimize the number of operations used, we then concentrated most of the multiplications to the first stage and performed them at once. As a result, the proposed scheme shows better PSNR characteristic than other DCT domain methods while it requires almost the same number of operations and memory.