How Do We Know What We Know?

A Field Guide to the Study Designs Behind Every Health Headline

My uncle once told me, with total confidence, that he knew a man who smoked two packs a day and lived to 97. Therefore, he concluded, cigarettes were probably fine. I was in graduate school at the time, and I made the mistake of trying to explain why one long-lived smoker is not evidence about smoking. I lost. You cannot out-argue a good story with a bad one, and “my buddy Javier” is a very good story.

That dinner is the reason for this whole series. Because underneath every health headline you have ever read — like coffee is good for you, coffee is bad for you, coffee will apparently do both by next Tuesday — there is a study. And not all studies are built the same. Some are quick snapshots. Some follow people for decades. Some are careful experiments, and some are just Javier.

Over the next several posts, I want to walk you through the main types of research studies, one at a time, from the humble to the heavyweight. My promise: no equations, no jargon I don’t translate, and at least one bad joke per post (I make no promises about quality).

Why Is the Design the Whole Ballgame?

Here is the single most useful idea I can give you: the type of study determines the type of claim it can support. A study that measures everyone once can tell you how common a disease is, but not what caused it. A study that randomly assigns people to a treatment can make a strong claim about cause, but it might be too small, too short, or too artificial to tell you what happens in the real world. Every design trades something away to get something else.

So when a friend sends you a scary headline, the sharpest question isn’t “Is this true?” It’s “What kind of study is this, and what is it actually allowed to say?” Answer that, and you’ve done most of the work of thinking clearly about health.

The evidence “staircase”: each design tends to support a sturdier claim than the one below it… Though a well-built lower step often beats a shaky higher one.

The Staircase of Evidence

Researchers often picture study designs as a pyramid or a staircase, with weaker designs near the bottom and stronger ones near the top. It’s a useful mental model, as long as you don’t treat it as gospel. Here’s the tour we’re about to take, from the ground floor up:

• Cross-sectional studies. These are a “photograph” or “snapshot” of a population at one moment. Great for counting how common something is (i.e., prevalence).

• Ecological studies. These compare whole groups or countries rather than individuals. They’re fast, cheap(ish) to do, wide-angle, and easy to misread.

• Case reports and case series. These are the first, careful descriptions of something new or strange. The alarm bell, not the diagnosis.

• Case-control studies. These studies start with an outcome and work backward to find likely causes. The detective’s design we use so much in outbreak investigations.

• Cohort studies. These studies follow groups forward in time to see who develops what. The long-haul design that can be expensive and complicated to do

• Randomized controlled trials. You might have heard of these as the “gold standard.” Spoiler alert: they’re not. Anyway, they’re the experiment where a coin flip decides who gets the treatment and who gets not the treatment. The gold standard (again, maybe?) for cause and effect.

• Systematic reviews. These have researchers gathering every decent study on a question and summarizing them honestly… Or as honestly as possible. (Humans are fallible, after all.)

• Meta-analyses. These are about mathematically pooling those studies into one combined, more powerful answer. The top of the staircase.

The one thing to remember

Higher on the staircase usually means a stronger claim about cause and effect, but a beautifully run cohort study beats a sloppy trial, and a huge meta-analysis of weak studies is still built on weak studies. Design matters. So does execution.

A Word of Comfort Before We Start

You do not need a degree in statistics to follow along. You already reason about evidence every day when you decide whether a restaurant review is trustworthy or whether one bad Uber ride means the driver is dangerous or you were just unlucky. Epidemiology is that same instinct, dressed up with better vocabulary and a lot more caffeine. (SO MUCH CAFFEINE!!!)

By the end of this series, my hope is that you’ll read a health headline and instinctively ask the right questions, and maybe, just maybe, be able to gently handle your own version of Uncle Javier.

Next in the series: we start on the ground floor with the cross-sectional study: the snapshot that can tell you how common something is, but will happily mislead you about why.