Characteristic polynomials of random unitary matrices have been intensively studied in recent years: by number theorists in connection with Riemann zetafunction, and by theoretical physicists in connection with Quantum Chaos. In particular, Haake and collaborators have computed the variance of the coefficients of these polynomials and raised the question of computing the higher moments. The answer turns out to be intimately related to counting integer stochastic matrices (magic squares). Similar results are obtained for the moments of secular coefficients of random matrices from orthogonal and symplectic groups. Combinatorial meaning of the moments of the secular coefficients of GUE matrices is also investigated and the connection with matching polynomials is discussed.