Researchers investigating personalization techniques for Web Information Retrieval face a challenge; that the data required to perform evaluations, namely query logs and clickthrough data, is not readily available due to valid privacy concerns. One option for researchers is to perform a user study, however, such experiments are often limited to small (and sometimes biased) samples of users, restricting somewhat the conclusions that can be drawn. Alternatively, researchers can look for publicly available data that can be used to approximate query logs and click-through data. Recently it has been shown that the information contained in social bookmarking (tagging) systems may be useful for improving Web search. We investigate the use of tag data for evaluating personalized retrieval systems involving thousands of users. Using data from the social bookmarking site del.icio.us, we demonstrate how one can rate the quality of personalized retrieval results. Furthermore, we conduct experimen...