Minimum spanning tree algorithms have been proposed for the lossy compression of image sets. In these algorithms, a complete graph is constructed from the entire image set and an average image, and a minimum spanning tree is used to determine which difference images to encode. In this paper, we propose a hierarchical minimum spanning tree algorithm in which the minimum spanning tree algorithm is first applied to clusters of similar images and then it is applied to the average images of the clusters. It is shown that the new algorithm outperforms the previous image set compression algorithms for image sets which are not very similar, especially at lower bitrates. Furthermore, the computational requirement for a minimum spanning tree is significantly lower than the previous minimum spanning tree algorithms.