Motion confidence measures aim to identify how well an image patch determines image motion. These kinds of confidence measures are commonly used to select points for optical flow estimation and feature tracking. The brute force approach of computing confidence for every image patch is inefficient, especially when the patches are large. The faster approach of computing confidence for a regular grid of patches is sub-optimal; good patches may be missed because they straddle grid boundaries. We present a new algorithm that efficiently selects globally optimal patches. Our primary innovation is the use of confidence bounds to identify image regions that should be explored by a queue-based search algorithm.