This paper describes a robust tracking algorithm for real-time, video based motion capture systems. Conventional motion capture systems are unable to capture motion data in real time because they use many video cameras and take a long time to deal with many images. To deal with problem, the authors have proposed in their earlier work [I] a real-time motion capture system using one video camera. It takes a video image of the upper part of the body of a person, and generates upper body motion data, e.g., x, y, z position of hands and a face rotation in real time and it employs a very simple tracking algorithm, but it suffers from occlusion problem. In this paper, the authors propose a more robust tracking algorithm that solves the occlusion problem. This paper mainly describes this algorithm, and delineates its usefulness by showing its application and experimental results.