Conservative adjustment of permutation p-values when the number of permutations is limited

The permutation procedure is widely used to assess the significance level (p-value) of a test statistic. This approach is asymptotically consistent. In genomics and proteomics studies, p-values are required to be evaluated at a “tiny” level. However, due to small sample sizes or limited computer resources, only a limited number of permutations can be obtained. Therefore, it is necessary to understand the accuracy of these permutation p-values. In this study, we show through the theory of order statistics that a considerable proportion of p-values will be under-evaluated by the permutation procedure. To solve this problem, we propose to conservatively adjust permutation p-values. The adjustment requires no parametric assumption on the distribution of test statistic. The solution can be expressed by a normalized incomplete beta function. The related normal distribution approximation is also discussed. Simulations are conducted to illustrate the proposed method and two microarray ge...

• Type: Journal
• Year: 2007
• Where: IJBRA
• Authors: Yinglei Lai
• Keywords: IJBRA 2007, P-values, Permutation, Permutation Procedure