The Gaussian mixture model is a powerful statistical tool in data modeling and analysis. Generally, the EM algorithm is utilized to learn the parameters of the Gaussian mixture. However, the EM algorithm is based on the maximum likelihood framework and cannot determine the number of Gaussians for a sample data set. In order to overcome this problem, we propose a new model selection criterion based on the kurtosis and skewness of the estimated Gaussians. Moreover, a new greedy EM algorithm is constructed via the kurtosis and skewness based criterion. The simulation results show that the proposed model selection criterion is efficient and the new greedy EM algorithm is feasible.