In this paper, we consider the optimal rate and power allocation that maximizes a general utility function of average user rates in a fading multiple-access or broadcast channel. By exploiting the greedy structure of the capacity-achieving resource allocation for both multiple-access and broadcast channels, it is established that utility-maximizing allocation policy can be obtained through dual-based gradient descent iterations with fast convergence and low complexity per iteration. Relying on stochastic averaging tools, we further develop a class of stochastic gradient iterations which are capable of asymptotically converging to the optimal benchmark with guarantees on the minimum average user rates, even when the fading channel distribution is unknown a priori.