Most methods for multiple camera tracking rely on accurate calibration to associate data from multiple cameras. However, it often is not easy to have an accurate calibration in some real applications due to practical reasons. The inaccurate calibration can lead to wrong data association of objects between cameras. In this paper, we propose a method to handle the data association of objects in multiple cameras under inaccurate ground plane homography by using the RFS Bayes filter. Our method is based on modeling measurements from cameras to a random finite set. This random finite set includes the primary measurement from the object, extraneous measurements of the object, and clutter. Experimental results show the efficiency and robustness of our method through challenging cases such as occlusions, merged and split persons.