Camera handoff is an important problem when using multiple cameras to follow a number of objects in a video network. However, almost all the handoff techniques rely on a robust tracker. State-of-the-art techniques used to evaluate the performance of camera handoff use either annotated videos or simulated data, and the handoff performance is evaluated in conjunction with a tracker. This does not allow a deeper understanding into the performance of a tracker and a handoff technique separately in the realworld settings. In this paper, we evaluate three camera handoff techniques, two different color-based trackers in seven real-life cases, with varying numbers of cameras, number of objects and the changing environmental conditions. We also perform experiments on annotated videos to provide the ground-truth for all the scenarios. This evaluation of performance isolates the effect of tracking and handoff techniques and clarifies their role in a video network.