This paper presents a simple mathematical analysis of some features of the Harmony Search algorithm (HS). HS is a recently developed derivative-free optimization algorithm, which draws inspiration from the musical process of searching for a perfect state of harmony. This work analyses the evolution of the population-variance over successive generations in HS and thereby draws some important conclusions regarding the explorative power of HS. Experimental results have been provided to validate the theoretical treatment. A simple modification of the classical HS has been proposed in the light of the mathematical analysis undertaken here.