Using statistics wrong is worse than not using it at all — you can reach a false conclusion with full confidence. Here are the main rakes everyone steps on. Each comes with how to avoid it and where to dive deeper.
Correlation ≠ causation
trap #1
Two things moving together doesn't mean one causes the other. A third factor — or coincidence — may be to blame. Only a randomized experiment proves causation.
The usual mechanism is a confounder (a shared cause): ice-cream sales and drownings rise together, but heat is the culprit, not ice-cream. Sometimes it's pure coincidence when you scan thousands of series. Before saying "it causes", ask: what else changes at the same time? Can you run an experiment (A/B) where you change only one factor yourself?
Misreading the p-value
trap
A p-value is not "the probability the hypothesis is true" nor "the chance it's random". It's the probability of a result this extreme (or more) if there were no effect. Confusing this is the source of most false claims.
How to avoid: say the correct wording out loud until it's automatic.
A small p says only: "if there were no effect, data like this would be rare". It does not measure the size of the effect or guarantee the hypothesis is true. p=0.04 with a broken design is worthless. Always read it together with the effect size and a confidence interval.
p-hacking & multiple testing
trap
Test 20 hypotheses at a 0.05 threshold and, on average, one will "fire" by chance. Cycling through metrics until something turns "significant" is self-deception.
How to avoid: fix the hypothesis in advance; with many tests apply a correction (Bonferroni, etc.).
Every test is a lottery ticket: a 5% chance of a false positive. 20 tickets ≈ a 64% chance of at least one "significant" result from pure noise. So decide what you're testing before you look at the data, and honestly report how many tests you ran.
Sampling bias
trap
If the sample isn't representative, the conclusion is wrong no matter how you crunch it. The classic is "survivorship bias": they looked only at planes that returned and reinforced the wrong spots.
How to avoid: ask "who/what is missing from this data?" before you analyse.
▸ dive deeper
Data is almost always collected through a filter: survivors, survey responders, active users. A conclusion holds only for that group, not for everyone. Abraham Wald argued to reinforce exactly where returning planes had no holes — the ones hit there never made it back.
Simpson's paradox
trap
A trend in the whole can reverse inside every subgroup. The aggregate lies when you mix different groups.
How to avoid: always re-check a conclusion split by key segments.
The classic is Berkeley admissions: overall, men were admitted more than women, yet within each department the opposite held — women simply applied more to competitive departments. The cure: before trusting an overall number, split by the obvious groups and check the conclusion survives.
Significant ≠ important
trap
On a large sample even a tiny difference becomes "statistically significant". Significant ≠ important for the business.
How to avoid: look at the effect size and confidence interval, not just p.
The p-value is sensitive to sample size: with millions of rows a 0.1% difference nobody cares about becomes "significant". Ask not "is it significant?" but "how big is the effect, and is it worth acting on?". That's what effect size (Cohen's d, difference in proportions) and a confidence interval are for.
Mean on skewed data
trap
On skewed data (incomes, session time) rare giants drag the mean up. One Bezos walks into a bar — and the "average" patron is a billionaire.
How to avoid: on skewed data use the median and percentiles, not the mean.
The mean is fine only for roughly symmetric data. Money, durations, city sizes have a long tail — there the median (50th percentile) describes the "typical" more honestly, and p90/p99 show the tail. Always draw a histogram before trusting a mean.
Small sample
trap
On a small sample percentages jump around at random: 3 of 4 = 75%, but it means nothing. The smaller the N, the wider the confidence interval.
How to avoid: report an estimate with an interval ("12% ± 4%") and don't conclude from a dozen observations.
Precision grows with the square root of N: to halve the interval you need four times the data. On small N a difference between groups is almost always "within noise". Before celebrating +20%, count how many observations are behind it.