This paper investigates the applicability of existing dialogue act annotation schemes, designed for the analysis of spoken dialogue, to the semantic annotation of multimodal data, and the way a dialogue act annotation scheme can be extended to cover dialogue phenomena from multiple modalities.