For regular, sparse, linear systems, like those derived from regular grids, using High Performance Fortran (HPF) for iterative solvers is straightforward. However, for irregular matrices the e cient implementation of solvers in HPF becomes much harder. First, the locality in the computations (a good partitioning) is unclear. Second, for e ciency we often use storage schemes that obscure even the simplest structure in the matrix (like rows and columns). Third, the limited capabilities of HPF to distribute data structures make it hard to implement the desired distribution. Fourth, data structures often have very di erent sizes and shapes, and matching the distributions for e cient implementation (locality) is a problem. Fifth, after implementing the distributions, we still must write the program in such a way that the compiler recognizes the e cient implementation and leaves out unnecessary communication, synchronization, etc. We discuss techniques for handling these problems, and our re...