The p-value and statistical significance
A p-value is the probability of a result at least as extreme as the one observed, assuming the null hypothesis of no real difference is true.
Remember
A p-value shows how unusual the data would be if the null hypothesis were true. It is not the probability that the null hypothesis is true, and it does not show that a result is clinically important.
Memory aid
A courtroom comparison may help: the null hypothesis is the presumption of innocence, and the p-value asks how surprising the evidence would be if the defendant really were innocent.
It keeps the direction straight: the p-value does not tell you how likely innocence is.
A hypothesis test begins with the null hypothesis, the starting assumption that there is no real difference or effect, such as a new drug working no better than placebo. The p-value then asks how surprising your data would be if that assumption were true. "At least as extreme" means as far from "no difference" as the result you saw, or further away.
A small p-value means the data would be unusual under the null hypothesis, so researchers doubt it and may reject it. By convention, a p-value below 0.05 is called statistically significant. That threshold is a convention and not a fact. The cut-off, often called alpha, is the significance level, set before the analysis.
- The p-value is not the probability that the null hypothesis is true. The p-value is a statement about the data, calculated on the assumption that the null hypothesis is true, and not about the hypothesis itself.
- The p-value does not show clinical importance. A very large trial can find a tiny, clinically irrelevant difference highly significant, so judge importance from the size of the effect and its confidence interval (the range around the estimate that shows how precise it is).
- The p-value is not the chance that a significant finding is a false positive. The chance that a "significant" finding is false depends on prior probability (how plausible the effect was before the study), sample size and the number of tests performed.
Take a trial where systolic blood pressure (the higher figure in a blood pressure reading) falls 1 mmHg more with the drug than with placebo, and p is 0.001. If the two treatments truly did not differ, a result this big or bigger would occur only about 0.1 percent of the time. A 1 mmHg fall is tiny.
Key facts
- Definition
- the probability that a result this extreme, or more extreme, would occur if the null hypothesis were true.
- Small p-value
- the data would be unusual if the null hypothesis were true, so researchers may reject it.
- Threshold
- a p-value below 0.05 is conventionally called statistically significant, but 0.05 is a convention, not a fact.
- Significance level
- fixed before the study, usually 0.05, and not the same thing as the p-value.
- Evidence, not proof
- a significant result argues against the null hypothesis without proving it false.
- Importance
- statistical significance says nothing about whether an effect is big enough to matter, so look at the effect size and confidence interval.
- Worked example
- with a p-value of 0.03, a difference this large or larger would occur about 3 percent of the time if the drug truly did nothing.
Exam traps
- A p-value of 0.001 is not a 0.1% probability that the drug has no true effect, because the p-value is worked out by assuming the null hypothesis is true.
- A p-value of 0.03 does not mean a 97 percent probability that the alternative hypothesis (that a real difference exists) is correct, because taking one minus the p-value is a common fallacy.
- A significant result does not prove the null hypothesis false, because it is evidence against it and data this extreme can still occur when it is true.
Read the full notes
Tested in 11 SimpleMed+ questions
Practice it in Pharmacology, with the answer and a full explanation after every question.
Practice these questions