Registration of CT and PET thoracic images has to cope with deformations of the lungs during breathing. Possible tumors in the lungs usually do not follow the same deformations, and this should be taken into account in the registration procedure. We show in this paper how to introduce tumor-based constraints into a non-linear registration of thoracic CT and PET images. Tumors are segmented by means of a semi-automatic procedure and they are used to guarantee relevant deformations near the pathology. Results on synthetic and real data demonstrate a significant improvement of the combination of anatomical and functional images for diagnosis and for oncology applications.