Non-Rigid image registration has been widely developed over the last years. However, many registration techniques do not take into account any a priori information on the structures in the images. We present in this article a general locally affine registration framework, which allows us to register local areas in the images using affine transformations having few degrees of freedom. Thanks to our novel polyaffine framework and Log-Euclidean regularization, we ensure a smooth, coherent and invertible transformation all over the image. Remarkably, this is achieved very efficiently, even in 3D. We illustrate our method with two applications: bone registration in the lower abdomen area and critical brain structures registration.