Visual Generalization in Programming by Example

In Programming by Example [PBE, also sometimes called "Programming by Demonstration"] systems, the system records actions performed by a user in the interface, and produces a generalized program that can be used later in analogous examples. A key issue is how to describe the actions and objects selected by the user, which determines what kind of generalizations will be possible. When the user selects a graphical object on the screen, most PBE systems describe the object using properties of the underlying application data. For example, if the user selects a link on a web page, the PBE system might represent the selection based on the link's HTML properties. In this article, we explore a different, and radical, approach -- using visual properties of the interaction elements themselves, such as size, shape, color, and appearance of graphical objects -- to describe user intentions. Only recently has the speed of image processing made feasible real-time analysis of screen im...

• Type: Journal
• Year: 2000
• Where: CACM
• Authors: Robert St. Amant, Henry Lieberman, Richard Potter, Luke S. Zettlemoyer
• Keywords: CACM 2000, Graphical Objects, User, Visual