Content-based image retrieval systems still have difficulties to bridge the semantic gap between the low-level representation of images and the high level concepts the user is looking for. Relevance feedback methods deal with this problem using labels provided by users, but only during the current retrieval session. In this paper, we introduce a semantic learning method to manage user labels in CBIR applications. Our approach uses a kernel matrix to represent semantic information in a statistical learning framework. The kernel matrix is updated according to labels provided by users after retrieval sessions. Experiments have been carried out on a large generalist database in order to validate our approach.