How does multiple testing correction work

For example, if you plan to perform a collection of follow-up experiments and are willing to tolerate having a fixed percentage of those experiments fail, then FDR analysis may be appropriate. Alternatively, if follow-up will focus on a single example, then the Bonferroni adjustment is more appropriate. It is worth noting that the statistics literature describes a related probability score, known as the 'local FDR' 7. Unlike the FDR, which is calculated with respect to a collection of scores, the local FDR is calculated with respect to a single score.

The local FDR is the probability that a particular test gives rise to a false positive. In many situations, especially if we are interested in following up on a single gene or protein, this score may be precisely what is desired. However, in general, the local FDR is quite difficult to estimate accurately.

Furthermore, all methods for calculating P -values or for performing multiple testing correction assume a valid statistical model—either analytic or empirical—that captures dependencies in the data. For example, scanning a chromosome with the CTCF motif leads to dependencies among overlapping nt sequences.

Also, the simple null model produced by shuffling assumes that nucleotides are independent. If these assumptions are not met, we risk introducing inaccuracies in our statistical confidence measures. In summary, in any experimental setting in which multiple tests are performed, P -values must be adjusted appropriately. The Bonferroni adjustment controls the probability of making one false positive call. In contrast, false discovery rate estimation, as summarized in a q -value, controls the error rate among a set of tests.

In general, multiple testing correction can be much more complex than is implied by the simple methods described here. In particular, it is often possible to design strategies that minimize the number of tests performed for a particular hypothesis or set of hypotheses.

For more in-depth treatment of multiple testing issues, see reference 8. Phillips, J. Cell , — Article Google Scholar. Kim, T. Staden, R. Methods Mol. Benjamini, Y. Google Scholar. Kerr, K. Bioinformatics 25 , — Storey, J. A Stat. Efron, B. Dudoit, S. Book Google Scholar. Schneider, T. Nucleic Acids Res. Download references. William S. You can also search for this author in PubMed Google Scholar. Correspondence to William S Noble.

Reprints and Permissions. Noble, W. How does multiple testing correction work?. Nat Biotechnol 27, — Download citation. Issue Date : December Anyone you share the following link with will be able to read this content:.

Sorry, a shareable link is not currently available for this article. Provided by the Springer Nature SharedIt content-sharing initiative. Microbiome Genome Biology BMC Public Health Cancer Cell International Translational Psychiatry Advanced search. Skip to main content Thank you for visiting nature. How does multiple testing correction work? Download PDF. You have full access to this article via your institution. Figure 1: Associating confidence measures with CTCF binding motifs scanned along human chromosome Full size image.

Correcting for multiple hypothesis tests. Costs and benefits help determine the best correction method. References 1 Phillips, J. Article Google Scholar 2 Kim, T. Google Scholar 5 Kerr, K. Article Google Scholar 7 Efron, B. Article Google Scholar 8 Dudoit, S.

Book Google Scholar 9 Schneider, T. Author information Affiliations William S. Rights and permissions Reprints and Permissions. About this article Cite this article Noble, W. Copy to clipboard. Greve , Christen L. Mumaw , Evan J. If your p-value is less than your selected alpha level typically 0.

If the p-value is above your alpha value, you fail to reject the null hypothesis. Begin typing your search term above and press enter to search. Press ESC to cancel. Skip to content Home How does multiple testing correction work?

Ben Davis June 1, How does multiple testing correction work? Why performing a Bonferroni correction will reduce power? What is p value correction? What is a corrected P value? What is FDR correction? What is FDR value? What is FDR in gene expression?

What is P-value and Q-value? How is benjamini Hochberg calculated? What is a good false discovery rate? What is local FDR? How do you find the p-value in statistics? What is p-value in plain English? What does high P-value mean? What does nominal P-value mean? What does P-value mean in regression?

Can you have a negative p-value? Is P value always positive?