In this article we present an automatic on-line color calibration system that makes extensive use of the spatial relationships between color classes in the color space. First, we introduce the definition of class-relative color spaces, where classes are represented in terms of their spatial relation to a base color class. Then, using class-relative color spaces, the system is able to remap classes from the already trained ones, which gives a starting point for training the remaining classes. The color-calibrating system also uses a feedback from the detected objects using the remapped (or partially trained) classes. As a result, the system is able to generate a complete color look-up table from scratch, and to adapt quickly to severe lighting condition changes. A particularity of our system is that it does not need to solve the natural ambiguity in color classes’ intersections, but it is able to keep and use it during color segmentation using the concept of soft-colors.