In this paper, we propose a framework for predicting the performance of a vision algorithm given the input image or video so as to maximize the algorithm's ability to provide the desired output. This is achieved by learning a relationship between the algorithm's behavior characteristics and the quality of the input. In general, each algorithm has the ability to provide a successful output dependent on the input characteristics. Unfortunately, the acceptable input variability for each algorithm is not known a priori. Nonetheless, this can be modeled through effective assessment of input image/video quality and the vision algorithm's response to the input. The key benefit of such a model is in its ability to predict or select one of many algorithms given a new input so that the probability of achieving the desired output is maximized without the need for executing all available algorithms. Our proposed framework models the performance prediction process as one that accoun...