Publication bias and funnel plots
Publication bias means studies with positive (favourable) results are published more often than those with null results, which show no clear effect, so published evidence can overstate a treatment's benefit.
Remember
Publication bias means positive results are published more often than negative or null ones, so published evidence tends to overstate a treatment's effect.
Memory aid
disappointing trials left in a desk drawer
Picture the disappointing trials left in a desk drawer, because the published pile alone looks better than the whole truth.
In publication bias, whether a study is published depends on its results. Studies with favourable findings, usually ones that are statistically significant (unlikely to be due to chance alone), tend to be published more often than studies with negative or null findings, meaning no clear effect or no difference between treatments. The published studies alone therefore look better than the whole truth.
A meta-analysis combines the results of several studies into one overall (pooled) estimate. It can only combine the studies that can be found. When the missing studies are mostly the disappointing ones, the pooled result tends to look better than the treatment really is. Missing evidence does not distort every meta-analysis in the same way, but the risk is real.
In a funnel plot, each dot is a study, placed by its effect estimate (the effect that study found) along the horizontal axis and by its size up the vertical axis. Large studies give precise results, so they sit near the top and cluster around the true effect, whereas small studies scatter more widely lower down, since chance affects them more. Without bias, the scatter on either side of the true effect is roughly even, giving a symmetrical funnel (the same on both sides).
If small studies that found little or no benefit are never published, one lower corner of the funnel, on the no-benefit side, is empty and the plot becomes asymmetrical, meaning lopsided. The pooled estimate then tends to overestimate the benefit of the treatment. Even so, asymmetry is a warning sign rather than proof, because genuine differences between trials, flawed small trials or chance can produce a similar shape, and a symmetrical plot does not rule publication bias out. Registering trials in advance lets planned studies that never appear be spotted.
Key facts
- Definition
- the chance of a study being published depends on what it found.
- Direction
- studies with positive results are published more often, so pooled results tend to overstate the benefit.
- Funnel plot axes
- each study's effect runs along the horizontal axis and its size up the vertical axis, so the smallest studies spread widest at the bottom.
- Pattern
- an empty bottom corner on the no-benefit side suggests that small null studies are missing.
- Caution
- asymmetry can have other causes, so it is a signal to investigate rather than proof.
- Safeguards
- registering trials in advance and searching for unpublished studies, and, with caution, using funnel plots to look for missing studies.
Exam traps
- A published study that presents only its statistically significant outcomes shows selective outcome reporting, not publication bias, because publication bias concerns whole studies that never appear.
- When the person enrolling the next patient can predict which group they will join, the result is selection bias inside a trial, not publication bias, because it has nothing to do with which finished studies reach print.
- Publication bias is not a preference for larger studies: it depends mainly on the direction or statistical significance of the results rather than on how big the study is.
Read the full notes
Tested in 6 SimpleMed+ questions
Practice it in Pharmacology, with the answer and a full explanation after every question.
Practice these questions