In this paper, we present a monocular 3D arm movement tracking system using adaptive particle filter. The effective sample size (ESS) is analyzed in the adaptive particle filter to tackle the abrupt dynamic changes of the arm movement. Sample-efficiency-optimized auxiliary particle filter (SEOAPF) is invoked when low ESS is detected. In SEO-APF, the auxiliary variable weights are computed to minimize the true importance weight variance, so the tracking results and the efficiency of the particle filters are improved. Experimental results have demonstrated the efficacy of this approach for 3D arm movement tracking.