Time varying environments or model selection problems lead to crucial dilemmas in identification and control science. In this paper, we propose a modular prediction scheme consisting in a mixture of expert architecture. Several Kalman filters are forced to adapt their dynamics and parameters to different parts of the whole dynamics of the system. The performances of this modular learning scheme are evaluated on a visual servoing problem: motion prediction of an object in a 3-D space for pursuing it with a 3 degree-of-freedom robot manipulator.