We present an effective method to optimize over the parameters of an image patch descriptor to obtain one that is computationally more efficient while maintaining a high recognition rate. We formulate the optimization problem in a multi-objective manner, which balances two conflicting goals while removing the need for traditional weighting coefficients. To this end we introduce the Pareto efficiency criterion, which helps finding solutions that increase one objective without decreasing the other. Despite the vast size of the search space, we show how a state-of-the-art Genetic Algorithm can be tailored to find good solutions. Not only does the resulting descriptor perform better than state-of-the-art ones, but our approach is of broader significance as optimization problems with balanced goals are often encountered in Computer Vision.