We propose a layered quality adaptation scheme for video streams to smooth the short-term oscillations induced by Additive Increase Multiplicative Decrease (AIMD) mechanisms, and eventually refine the perceptual video quality. The layered scheme utilizes receiver buffering, adapting the video quality along with long-term variations in the available bandwidth. The allocation of a new layer is based on explicit criteria that consider the available bandwidth, as well as the amount of buffering at the receiver. Consequently, the adaptation mechanism prevents wasteful layer changes that have an adverse effect on userperceived quality. In the sequel, we concentrate on the interactions of the layered approach with Scalable Streaming Video Protocol (SSVP). Exploiting performance measures related to the perceived quality of rate-adaptive video streams, we quantify the combination of SSVP rate control and receiverbuffered layered adaptation.