This paper describes a method for articulated upper body tracking in monocular scenes. The compatibility between model and the image is estimated using one particle filter for each limb and the compatibility between limbs is represented by interaction potentials. The joint probability is obtained by belief propagation on a factor graph. The body model is a loose limbed model including attraction potentials between adjacent limbs and constraints to reject poses resulting in collisions. Robust compatibility functions based on face color, edges and motion energy are used to evaluate the likelihood of the generated hypotheses. Experimental results show the upper body tracking efficiency of the proposed algorithm.