This paper studies the effect of discretizing the parametrization of a dictionary in the particular context of Matching Pursuit decompositions of signals. Our approach relies on the observation that the continuously parametrized dictionary can be seen as an embedded manifold in the signal space on which the tools of differential (Riemannian) geometry can be applied. The main contribution of this paper is twofold. First, we prove theoretically that if a discrete dictionary reaches a minimal density criterion, then, the corresponding discrete MP (dMP) is equivalent in terms of convergence to a weakened hypothetical continuous MP. The weakness factor is precisely linked to a density measure of the discrete dictionary. Second, we show that the insertion of a simple geometric gradient ascent optimization on the atom dMP selection maintains the previous comparison but with a weakness factor at least two times closer to unity than without optimization. These theoretical results are finally ex...