It is difficult to navigate virtual environment as in real world and to interact with other participant in virtual environment, especially wearing Head-Mounted Display (HMD). We developed Virtual Office Environment System (VOES), and avatar is used to navigate and to interact with other participants. For easy and intuitive control of avatar motion in the system, we use continuous hand gesture recognition system. State automata are proposed in hand gesture recognition to segment continuous hand gesture and to remove meaningless motion. Using avatar and gesture interface, this system provides natural navigation and interaction in virtual environment system.