A new approach to false discovery rates and multiple hypothesis testing


Testing multiple hypotheses involves guarding against much more complicated errors than when testing a single hypothesis. Whereas one typically controls the Type I error rate for a single hypothesis test, the Family Wise Error Rate (FWER) or the False Discovery Rate (FDR) are controlled for multiple hypothesis tests. Therefore, just as in single hypothesis testing, the acceptable error rate is fixed and the rejection region is found to control the error rate. Controlling the FWER or FDR often involves complicated sequential p-value rejection methods based on the observed data. In other words, the rejection region is estimated from the data. In this paper we propose the opposite approach --- fix the rejection region and then estimate the error rate. This new approach offers increased applicability, accuracy, and power. We apply this methodology to the FDR and provide evidence for its benefits. Also discussed is the calculation of the q-value, which is the FDR analogue of the p-value. Some simple numerical examples are presented that show this new approach can yield over a 10 times increase in power compared to the Benjamini and Hochberg (1995) method. We also briefly discuss how this approach can be applied to other multiple hypothesis testing error measures, such as the FWER.

Stanford Statistics Technical Report