Improving acoustic event detection using generalizable visual features and multi-modality modeling

Acoustic event detection (AED) aims to identify both timestamps and types of multiple events and has been found to be very challenging. The cues for these events often times exist in both audio and vision, but not necessarily in a synchronized fashion. We study improving the detection and classification of the events using cues from both modalities. We propose optical flow based spatial pyramid histograms as a generalizable visual representation that does not require training on labeled video data. Hidden Markov models (HMMs) are used for audio-only modeling, and multi-stream HMMs or coupled HMMs (CHMM) are used for audio-visual joint modeling. To allow the flexibility of audio-visual state asynchrony, we explore effective CHMM training via HMM state-space mapping, parameter tying and different initialization schemes. The proposed methods successfully improve acoustic event classification and detection on a multimedia meeting room dataset containing eleven types of general non-spe...

• Type: Journal
• Year: 2011
• Where: ICASSP
• Authors: Po-Sen Huang, Xiaodan Zhuang, Mark Hasegawa-Johnson
• Keywords: Acoustic Event, Acoustic Event Detection, Audio-visual Joint Modeling, ICASSP 2011, Signal Processing