The performance of punctured LDPC codes under maximum-likelihood (ML) decoding is studied in this paper via deriving and analyzing their average weight distributions (AWDs) and the corresponding asymptotic growth rate of the AWDs. In particular, we prove that capacity-achieving codes of any rate and for any memoryless binary-input output-symmetric (MBIOS) channel under ML decoding can be constructed by puncturing some original LDPC code with small enough rate. Moreover, we prove that the gap to capacity of all the punctured codes can be the same as the original code with a small enough rate. Conditions under which puncturing results in no rate loss with asymptotically high probability are also given in the process. These results show high potential for puncturing to be used in designing capacity-achieving codes, and also be used in rate-compatible coding under any MBIOS channel. This work was supported in part by the National Science Foundation under Grant CAREER-CCF-0346977.