What are the differences between FDR, adjusted p value and q value? I saw this interesting answer by Gordon Smyth explaining the origins and corresponding implementations in various R packages.
The thing to understand is that terms like FDR and q-value were defined in specific ways by their original inventors but are used in more generic ways by later researchers who adapt, modify or use the ideas.
The term “false discovery rate (FDR)” was created by Benjamini and Hochberg in their 1995 paper. They gave a particular definition of what they meant by FDR. Their procedure accepted or rejected hypotheses, but did not produce adjusted p-values.
Benjamini and Yekutieli presented another more conservative algorithm to control the FDR in a 2001 paper. Same definition of FDR, but a different algorithm.
In 2002, I re-interpreted the Benjamini and Hochberg (BH) and Benjamini and Yekutieli (BY) procedures in terms of adjusted p-values. I implemented the resulting algorithms in the function p.adjust() in the stats package, and used them in the limma package, and this lead to the concept of an FDR adjusted p-value. The terminology used by the p.adjust() function and limma packages has lead people to refer to “BH adjusted p-values”.
The adjusted p-value definition that you give is essentially the same as the BH adjusted p-value, except that you omitted the last step in the procedure. Your definition as it stands is not an increasing function of the original p-values.
In 2002, John Storey created a new definition of “false discovery rate”. Storey’s definition is based on Benjamini and Hochberg’s original idea, but is mathematically a bit more flexible. John Storey also created the terminology “q-value” for a quantity that estimates his definition of FDR. He implemented q-value estimation procedures in an R package called qvalue.
Another important but often overlooked difference is the idea of FDR “estimation” vs FDR “control”. The qvalue package attempts to give a more or less unbiased estimate of the FDR, so the true FDR is about equally likely to be greater or less in practice. The BH approach instead controls the expected FDR. It guarantees that the true FDR rate will be less than the specified rate on average if you do an exactly similar experiment over and over again. So the BH approach is slightly more conservative than qvalue. The BH properties hold regardless of the number of p-values, while qvalue is asymptotic, so the BH approach is more robust than qvalue when the number of hypotheses being tested isn’t very large.
So, strictly speaking, the q-value and the FDR adjusted p-value are similar but not quite the same. However the terms q-value and FDR adjusted p-value are often used generically by the Bioconductor community to refer to any quantity that controls or estimates any definition of the FDR. In this general sense the terms are synonyms.
The lesson to draw from this is that different methods and different packages are trying to do slighty different things and give slightly different results, and you should always cite the specific software and method that you have used.