We developed a compact vision system for wearable interface applications using a vision chip. The vision chip is capable of real-time vision at higher frame rates than the video frame rate with a single chip. Using this system, vision-based wearable man-machine interfaces, such as a portable eye tracker and a six-degrees-of-freedom input device, can be realized at higher frame rates than ever before.