In this paper, we present a multimodal discourse ontology that serves as a knowledge representation and annotation framework for the discourse understanding component of an artificial personal office assistant. The ontology models components of natural language, multimodal communication, multi-party dialogue structure, meeting structure, and the physical and temporal aspects of human communication. We compare our models to those from the research literature and from similar applications. We also highlight some algorithms that are used to perform automatic processing and understanding using these models and suggest elements of the ontology that may be of immediate interest to meeting annotation by human or automated means.