This paper addresses the question whether and under which conditions hearers take into account the perspective of the speaker, and vice versa. Empirical evidence from computational modeling, psycholinguistic experimentation and corpus research suggests that a distinction should be made between speaker meanings and hearer meanings. Literal sentence meanings result from the hearer's failure to calculate the speaker meaning in situations where the hearer's selected meaning and the speaker meaning differ. Similarly, non-recoverable forms result from the speaker's failure to calculate the hearer meaning in situations where the speaker's intended meaning and the hearer meaning differ.