The nature of the P value has recent come up in the New England Journal of Medicine. Pocock, a statistician, is quoted as saying that “a P value of 0.05 carries a 5% risk of a false positive result.”
Such a statement is obviously wrong, and Daniel Hu complains, correctly, that it is a “misconception”. So Pocock and Stone reply that the p value of 0.05 carries a 5% risk of carrying a false positive result “when there is no true difference between treatments.” This is correct, provided it is understood that false positive does not mean that the probability that the treatment is not effective is 5%. When is it reasonable to suppose that there is absolutely no true difference between treatments? Hardly ever. So the P value is not very useful to decision makers. The CLAHRC WM Director cautions statisticians not to discount prior consideration of how likely/ realistic a null hypothesis is. Homeopathy aside, it is seldom a plausible prior hypothesis.
— Richard Lilford, CLAHRC WM Director
- Pocock SJ & Stone GW. The primary outcome is positive – is that good enough? N Engl J Med. 2016; 375: 971-9.
- Hu D. The Nature of the P Value. N Engl J Med. 2016; 375: 2205.
- Pocock SJ & Stone GW. Author’s Reply. N Engl J Med. 2016; 375: 2205-6.