Modelling and analyzing multimodal dyadic interactions using social networks

Social network analysis became a common technique used to model and quantify the properties of social interactions. In this paper, we propose an integrated framework to explore the characteristics of a social network extracted from multimodal dyadic interactions. First, speech detection is performed through an audio/visual fusion scheme based on stacked sequential learning. In the audio domain, speech is detected through clusterization of audio features. Clusters are modelled by means of an One-state Hidden Markov Model containing a diagonal covariance Gaussian Mixture Model. In the visual domain, speech detection is performed through differential-based feature extraction from the segmented mouth region, and a dynamic programming matching procedure. Second, in order to model the dyadic interactions, we employed the Influence Model whose states encode the previous integrated audio/visual data. Third, the social network is extracted based on the estimated influences. For our study, we u...

• Type: Journal
• Year: 2010
• Where: ICMI
• Authors: Sergio Escalera, Petia Radeva, Jordi Vitrià, Xavier Baró, Bogdan Raducanu
• Keywords: Biometrics, Dyadic Interactions, ICMI 2010, Model, Social Networks