Breast cancer is one of the most common diseases among women. Content-based mammogram retrieval has been proposed to aid various medical procedures. To develop a content-based mammogram retrieval system, textural feature extraction is one of the crucial requirements. This study proposes a Gabor filtering method for the extraction of textural features, which firstly performs Gabor filtering on the underlying image, applies the physical properties of a probability wave to probability transformation and then computes features to describe the textural pattern of the mammogram. This study also proposes an adaptive strategy for feature selection, filter selection and feature weighting, which utilizes a user’s relevance feedback to reduce the redundancy in the representation and incorporates the user’s information needs in image retrieval. Experimental results show that hypothesis tests can effectively find discriminated features and this retrieval system can improve its performance thro...