A Flexible Compute and Memory Infrastructure for High-Level Language to Hardware Compilation