The geometric features of images, such as edges, are difficult to represent. When a redundant transform is used for their extraction, the compression challenge is even more difficult. In this paper we present a new rate-distortion optimization algorithm based on graph theory that can encode efficiently the coefficients of a critically sampled, non-orthogonal or even redundant transform, like the contourlet decomposition. The basic idea is to construct a specialized graph such that its minimum cut minimizes the energy functional. We propose to apply this technique for rate-distortion Lagrangian optimization in subband image coding. The method yields good compression results compared to the state-of-art JPEG2000 codec, as well as a general improvement in visual quality.