The recent advances in non-scalable video encoding brought by the H.264/AVC standard offered significant improvements in terms of rate-distortion performance. This paper proposes a H.264/AVC based fine grain scalable video encoder which also exploits the motion compensation tools of the H.264/AVC standard to explore the temporal redundancy in the enhancement layer. The enhancement layer is predicted from a high quality reference obtained from past information of the enhancement and base layers. One of the drawbacks of this architecture is the drift effect, which occurs when part of the enhancement layer used for prediction is not received by the decoder. The drift reduction approaches here proposed simultaneously allow improvements in the coding efficiency and a reduction of the drift effect. The experimental results show improvements up to 2 dB in coding efficiency in comparison to Intra coding (like used by the MPEG-4 FGS standard) using the MPEG-4 testing conditions.