Understanding Natural Language with Diagrams

We describe a program, BEATRIX, that can understand textbook physics problems specified by a combination of English text and a diagram. The result of the understanding process is a unified internal model that represents the problem, including information derived from both the English text and the diagram. The system is implemented as two opportunistic coparsers, one for English and one for diagrams, within a blackboard architecture. A central problem is establishing coreference, that is, determining when parts of the text and diagram refer to the same object. Constraints supplied by the text and diagram mutually reduce ambiguity in interpretation of the other modality.

• Type: Conference
• Year: 1990
• Where: AAAI
• Authors: Gordon S. Novak, William C. Bulko
• Keywords: AAAI 1990, English Text, Intelligent Agents, Textbook Physics Problems, Unified Internal Model