In this paper, we propose a method of estimating camera parameters for annotating 3D virtual objects with a wearable vision system. The proposed method tracks natural feature points using a doubled color histogram matching and template matching. The model-view matrix calculation method consists of three algorithms for PnP problems and uses each algorithm selectively. The experimental system using the proposed method works at 5 to 10 frames per second.