The advantages of pattern-based programming have been well-documented in the sequential literature. However patterns have yet to make their way into mainstream parallel computing, even though several research tools support them. There are two critical shortcomings of pattern (or template) based systems for parallel programming: lack of extensibility and performance. The patterns supported by a tool are typically limited in number or scope, thereby narrowing the applicability of a given system. As well, patterns usually offer generic solutions, which incur additional runtime overhead that impacts performance. This paper describes our approach for addressing these problems in the CO2P3S parallel ing system. CO2P3S supports multiple levels of abstraction, allowing the user to design an application with high-level patterns, but move to lower levels of abstraction for performance tuning. Patterns are implemented as parameterized templates, allowing the user the ability to customize the pa...