The class of all natural images is an extremely small fraction of all possible images. Some of the structure of natural images can be modeled statistically, revealing striking regularities. Moreover, the human visual system appears to be optimized to view natural images. Images that do not behave statistically as natural images are harder for the human visual system to interpret. This paper reviews second order image statistics as well as their implications for computer graphics. We show that these statistics are predominantly due to geometric modeling, while being largely unaffected by the choice of rendering parameters. As a result, second order image statistics are useful for modeling applications, which we show in direct examples (recursive random displacement terrain modeling and solid texture synthesis). Finally, we present an image reconstruction filter based on second order image statistics.