This paper deals with understanding referring expressions involving perceptual grouping. The ability to use referring expressions is important for conversational agents aimed at real-world interaction. We conducted a psychological experiment to collect referring expressions involving perceptual grouping. A set of methods to identify referents based on the collected data are presented. We were able to identify 78.8% of the referents in the collected expressions.