We present a method for automatically learning a set of discriminatory facial components for face recognition. The algorithm performs an iterative growing of components starting with small initial components located around preselected points in the face. The direction of growing is determined by the gradient of the cross-validation error of the component classifiers. In experiments we analyze how the shape of the components and their discriminatory power changes across different individuals and views.