Multimodal knowledge capture from text and diagrams

Many information sources use multiple modalities, such as textbooks, which contain both text and diagrams. Each captures information that is hard to express in the other, and evidence suggests that multimodal information leads to better retention and transfer in human learners. This paper describes a system that captures textbook knowledge, using simplified English text and sketched versions of diagrams. We present experimental results showing it can use captured knowledge to answer questions from the textbook’s curriculum. Categories and Subject Descriptors I.2.4 Knowledge Representation Formalisms and Methods

• Type: Conference
• Year: 2009
• Where: KCAP
• Authors: Kate Lockwood, Kenneth D. Forbus
• Keywords: Information Retrieval, KCAP 2009, Knowledge Representation Formalisms And Methods, Many Information Sources, Textbook Knowledge