An analysis of steganographic systems subject to the following perfect undetectability condition is presented in this paper. Following embedding of the message into the covertext, the resulting stegotext is required to have exactly the same probability distribution as the covertext. Then no statistical test can reliably detect the presence of the hidden message. We refer to such steganographic schemes as perfectly secure. A few such schemes have been proposed in recent literature, but they have vanishing rate. We prove that communication performance can potentially be vastly improved; specifically, we construct perfectly secure steganographic codes from public watermarking codes using binning methods and randomization of the code over an invariant group associated with the covertext distribution (e.g., a permutation group in the case of independently and identically distributed covertext). We derive (positive) capacity and random-coding exponents for perfectly secure steganographic s...