Modeling user affect is becoming increasingly important for intelligent interfaces and agents that aim to establish believable interactions with their users. However, evaluating the accuracy and effectiveness of affective user models is extremely challenging because of the many unknowns in affect comprehension. In this paper, we overview existing approaches related to the validation of affective user models, and we describe our own experience with an approach for direct model evaluation that we have used in a recent study.