We investigate a computational model of the emergence of gaze following that is based on a generic basic set of mechanisms. Whereas much attention has been focused so far on the study of the infant’s behavior, we systematically analyze the caregiver and show that he plays a crucial role in the development of gaze following in our model, especially for infant models with simulated developmental disorders such as autism and Williams syndrome. We first create two reference infant parameter sets and test their behavior with a simple standard caregiver. Based on these findings we then propose new caregiver models and evaluate them on normally developing infants and on infants with simulated developmental disorders. Further, we investigate if and how a pair of infants (with and without simulated developmental disorders) might learn gaze following from scratch, without a mature caregiver. The findings of this paper suggest the pivotal role the caregiver plays for the infant in developin...