This paper proposes a selective procedure inlining scheme to improve a multi-grain parallelism, which hierarchically exploits the coarse grain task parallelism among loops, subroutines and basic blocks and near fine grain parallelism among statements inside a basic block in addition to the loop parallelism. Using the proposed scheme, the parallelism among different layers(nested levels) can be exploited. In the evaluation using 103.su2cor, 107.mgrid and 125.turb3d in SPEC95FP benchmarks on 16 way IBM pSeries690 SMP server, the multi-grain parallel processing with the proposed scheme gave us 3.65 to 5.34 times