Conventional digital video playback systems provide only limited user interactivity, mostly in the form of VCRlike controls such as fast forward/backward, slow motion, pause/stop, etc. In this model, the temporal ordering and the spatial viewpoints of the video streams being viewed are completely determined at authoring time. In contrast, we have defined a form of interactive video called active video [NEO01], which supports hyper-linking among related video sequences and interpolation of video sequences with neighboring viewpoints, to offer end users the additional flexibility of choosing the sequencing and the viewing angle (even virtual ones) at playback time, respectively. However, flexibility of spatial navigation in active video leads to a substantially higher storage and transmission cost due to multiple time-synchronized video streams capturing the same scene and the meta-data encoding their spatial association. In fact, since the meta-data cost is comparable to that of video ...