We propose a new theoretical framework for generalizing the traditional notion of covariance. First, we discuss the role of pairwise cross-cumulants by introducing a cluster expansion technique for the cumulant generating function. Next, we introduce a novel concept of symmetry decomposition of probability density functions according to the C4v group. By utilizing the irreducible representations, generalized covariances are explicitly defined, and their utility is demonstrated using an analytically solvable model.