The recent developments in image-based rendering have enabled a representation of virtual environments based on a simulation of panoramas, which we call virtual panoramas. Current virtual panorama systems do not provide a natural and immersive interaction with the environment. We propose a new system that uses hardware and software components to provide a natural and immersive interaction with virtual panoramas. As part of the system we propose a specific representation for the interactions in a virtual panorama. This representation can be used as a basis for the design of a high-level language for the creation of such environments.