Generalized linear mixed model with a penalized Gaussian mixture as a random effects distribution