Reading research critically

A study lands in a journal, so it must be true, right? Not so fast. Being a good consumer of research means reading with a skeptical eye — hunting for the flaws that can make even a published study misleading. Examiners want you to spot the classic weaknesses.

Common flaws to watch for: small or unrepresentative samples (findings that won't hold up or generalize); no control/comparison group (you can't credit the treatment without something to compare against); lack of random assignment (confounds sneak in); confounding variables the authors didn't control; poor measurement (unreliable or invalid instruments); and the cardinal error of confusing correlation with causation — two things moving together doesn't prove one caused the other. Also flag overgeneralized conclusions that stretch far beyond what the data support, and possible experimenter bias or conflicts of interest (e.g., who funded it?).

One systemic flaw deserves special attention: publication bias, sometimes called the "file-drawer problem." Studies with positive, statistically significant results are far more likely to get published, while null or negative findings get stuffed in a drawer. The result is a distorted literature that overstates how well treatments work — and it can even inflate the results of meta-analyses that only pool the published studies.

Clinical Pearl: Never treat "published" as "proven." The two flaws the NCE loves: correlation does not equal causation, and publication bias (positive results get published, null results get filed away) skews the literature toward overstating effects.