The goal of this research is to develop performance profiles of parallel and distributed applications in order to predict their execution time under different network conditions. This paper measures the resource requirements of the NAS benchmark programs and characterizes their performance in a shared heterogeneous environment. The programs in the benchmark suite were executed on a controlled testbed and their usage of CPU, bandwidth, and memory were measured. The performance of the benchmark programs was also measured under controlled sharing of CPU and bandwidth. The results are used to characterize the behavior of the NAS benchmark programs with resource sharing. The paper demonstrates that the core system activity of a program can be accurately measured by passive probing, and that this measured system activity is the key to the prediction of program performance when resources must be shared. Our methods rely on system level measurements alone, and therefore, application knowledg...