In this paper, we describe a novel statistical approach to the vocal tract transfer function (VTTF) estimation of a speech signal based on a factor analyzed trajectory hidden Markov model (HMM). Because speech is a quasi-periodic signal, there are many missing frequency components between adjacent F0 harmonics. The proposed method determines a time-varying VTTF sequence based on the maximum a posteriori (MAP) estimation considering not only harmonic components observed at each analyzed frame but also those at other frames for stochastically interpolating the missing frequency parts.