Smart camera is a camera that can not only see but also think and act. A smart camera is an embedded vision system which captures and processes image to extract application-specific information in real time. The brain of a smart camera is a special processing module that performs application specific information processing. The design of a smart camera as an embedded system is challenging because video processing has insatiable demand for performance and power, but at the same time embedded systems place considerable constraints on the design. We present our work to develop GestureCam, an FPGA-based smart camera built from scratch that can recognize simple hand gestures. The first completed version of GestureCam has shown promising realtime performance and is being tested in several desktop HCI (Human Computer Interface) applications.