In this paper we present an autonomous meeting summarizer that transcribes an agile planning meeting and produces a textual summary of the discussion. We explore the issues involved in designing a speech-based interactive system that communicates with humans in a natural language. The inherent nature of ambiguity in conversational speech is overcome by suggesting a list of possible phrases to listen for. The system interacts with users in an interview-style dialogue for data collections. This is possible because we used the highly constrained structure and terminologies of agile planning meetings to make the approach successful. Keywords Human-to-computer Speech Communication, Speechbased User Interface, Spoken Dialogue Summarizer ACM Classification Keywords H5.2. Information interfaces and presentation (e.g., HCI): User Interfaces ? Natural language.