This paper presents a robust tracking system for autonomous robots equipped with omnidirectional cameras. The proposed method uses a 3D shape and color-based object model. This allows to tackle difficulties that arise when the tracked object is placed above the ground plane floor. Tracking under these conditions has two major difficulties: first, observation with omnidirectional sensors largely deforms the target’s shape; second, the object of interest embedded in a dynamic scenario may suffer from occlusion, overlap and ambiguities. To surmount these difficulties, we use a 3D particle filter to represent the target’s state space: position and velocity with respect to the robot. To compute the likelihood of each particle the following features are taken into account: i) image color; ii) mismatch between target’s color and background color. We test the accuracy of the algorithm in a RoboCup Middle Size League scenario, both with static and moving targets.