In this paper, we present a new end-to-end protocol, namely Scalable Streaming Video Protocol (SSVP), which operates on top of UDP and is optimized for unicast video streaming applications. SSVP employs Additive Increase Multiplicative Decrease (AIMD)-based congestion control and adapts the sending rate by properly adjusting the inter-packet-gap (IPG). The smoothness-oriented modulation of AIMD parameters and IPG adjustments reduce the magnitude of AIMD oscillation and allow for smooth transmission patterns, while TCP-friendliness is maintained. Our experimental results demonstrate that SSVP eventually adapts to the vagaries of the network and achieves remarkable performance on real-time video delivery. In the event where awkward network conditions impair the perceptual video quality, we investigate the potential improvement via a layered adaptation mechanism that utilizes receiver buffering and adapts video quality along with long-term variations in the available bandwidth. The adap...