Three-dimensional appearance models consisting of spatially varying reflectance functions defined on a known shape can be used in analysis-by-synthesis approaches to a number of visual tasks. The construction of these models requires the measurement of reflectance, and the problem of recovering spatially varying reflectance from images of known shape has drawn considerable interest. To date, existing methodsrely on either:1) low-dimensional (e.g., parametric)reflectancemodels, or 2) large data sets involvingthousandsof images (or more) per object. Appearance models based on the former have limited accuracy and generality since they require the selection ofa specific reflectance model a priori, and while approaches basedon the latter maybe suitable forcertainapplications, theyare generally too costly and cumbersome to be used for image analysis. We present an alternative approach that seeks to combine the benefits of existing methods by enabling the estimation of a nonparametric spatial...