How social structure and institutional order co-evolve beyond instrumental rationality