Visual similarity matrices (VSMs) are a common technique for visualizing graphs and other types of relational data. While traditionally used for small data sets or well-ordered large data sets, they have recently become popular for visualizing large graphs. However, our experience with users has revealed that large VSMs are difficult to interpret. In this paper, we catalog common structural features found in VSMs and provide graph-based interpretations of the structures. We also discuss implementation details that affect the interpretability of VSMs for large data sets. CR Categories: I.3.6 [Computer Graphics]: Mehtodology and Techniques—Interaction techniques; G.4. [Mathematric Software]: User interfaces