Contours and surfaces are basic qualities which are processed by the visual system to aid the successful behavior of autonomous beings within the environment. There is increasing evidence that the two modalities of contours and surfaces are processed in separate, but interacting visual streams or sub-systems. Neurons at early stages in the visual system show strong responses only at locations of high contrast, such as edges, but only weak responses within homogeneous regions. Thus, for the processing and representation of surfaces, the visual system has to integrate sparse local measurements into a dense, coherent representation. We suggest a mechanism of confidence-based filling-in, where a confidence measure ensures a robust selection of sparse contrast signals. The new mechanism supports the generation of surface representations which are invariant against size and shape transformation. The filling-in process is controlled by contour or boundary signals which stop the filling-i...