We present the concept and motivations for the development of Affect Listeners, conversational systems aiming to detect and adapt to affective states of users, and meaningfully respond to users' utterances both at the content- and affect-related level. In this paper, we describe the system architecture and the initial set of core components and mechanisms applied, and discuss the application and evaluation scenarios of Affect Listener systems.