A Text-to-Picture Synthesis System for Augmenting Communication

We present a novel Text-to-Picture system that synthesizes a picture from general, unrestricted natural language text. The process is analogous to Text-to-Speech synthesis, but with pictorial output that conveys the gist of the text. Our system integrates multiple AI components, including natural language processing, computer vision, computer graphics, and machine learning. We present an integration framework that combines these components by first identifying informative and ‘picturable’ text units, then searching for the most likely image parts conditioned on the text, and finally optimizing the picture layout conditioned on both the text and image parts. The effectiveness of our system is assessed in two user studies using children’s books and news articles. Experiments show that the synthesized pictures convey as much information about children’s stories as the original artists’ illustrations, and much more information about news articles than their original photos alo...

• Type: Conference
• Year: 2007
• Where: AAAI
• Authors: Xiaojin Zhu, Andrew B. Goldberg, Mohamed Eldawy, Charles R. Dyer, Bradley Strock
• Keywords: AAAI 2007, Image Parts, Intelligent Agents, Natural Language, Natural Language Text