This paper proposes a Riemannian geometric framework to compute averages and distributions of point configurations so that different configurations up to affine transformations are considered to be the same. The algorithms are fast and proven to be robust both theoretically and empirically. The utility of this framework is shown in a number of affine invariant clustering algorithms on image point data.