Biostatistics: When Is a Difference Really a Difference?
How hypothesis testing and p-values help us think about random noise, and why statistical significance is not the same as real-world importance.
This is the fourth part of the 5-part series on biostatistics. Part 1 is here. Part 2 is here. Part 3 is here.
We have reached the moment the whole investigation has been pointing toward. When we split our interviewed attendees into those who ate the potato salad and those who did not, the potato salad group got sick far more often. Among potato salad eaters, the attack rate was high; among those who skipped it, it was low. The difference looks dramatic. But we just spent all of Part 3 learning that samples wobble. So the honest question is not “is there a difference?” There is always some difference between two groups, just by luck. The honest question is: Is this difference bigger than what luck alone would produce?
That question is what hypothesis testing was invented to answer, and answering it well is one of the most important skills in all of public health, partly because the most common tool for it, the p-value, is also the most widely misunderstood number in modern science.
