This paper introduces a new design to allow interactive, direct manipulation of volume data on volumetrically rendered images. We present an adaptive volume proxy mesh which serves not to define surfaces, but to encode the geometry and physical state of the volume. This system performs a modeling-free form of direct volume deformation by adaptively constructing the whole geometric structure on the background while keeping the structure transparent to the user. A complex, high-frequency structure is locally encoded into the mesh on-the-fly based on the user's volume of interest. This paper also presents a scheme for volume shading/shadowing applied in combination with our framework for improving reality in the visualization of time-varying deformable objects. We demonstrate examples of geometryencoded direct volume manipulation and application to volume exploration and surgical simulation.