Based in part on observations about the incremental nature of most state changes in biological systems, we introduce the idea of Memory with Memory in Genetic Programming (GP), where we use “soft” assignments to registers instead of the “hard” assignments used in most computer science (including traditional GP). Instead of having the new value completely overwrite the old value of the register, these soft assignments combine the old and new values. We then report on extensive empirical tests (a total of 12,800 runs) on symbolic regression problems where Memory with Memory GP almost always does as well as traditional GP, while significantly outperforming it in several cases. Memory with Memory GP also tends to be far more consistent, having much less variation in its best-of-run fitnesses than traditional GP. The data suggest that Memory with Memory GP works by successively refining an approximate solution to the target problem. This means it can continue to improve (if slow...