Hypothesis testing without the textbook

The last lesson left a rule of thumb dangling, half-justified: if the interval for a difference straddles zero, you have no winner. That instinct — “is this gap real, or just the noise the CLT told me to expect?” — is the right one, but an instinct is not a decision. You launch a new checkout flow. The new version converts at 4.3%, the old at 4.0%. Is the new one actually better, or did you draw a lucky sample? Hypothesis testing is the machinery that turns that question into a verdict with a controlled error rate — and it comes with caveats every data person eventually trips over.

The game: null vs alternative

You always start with a null hypothesis H₀ — the boring default you are trying to disprove (“nothing is happening here; any difference is pure chance”). Against it stands the alternative H₁ — the interesting claim, “something is happening.” For an A/B test:

• H₀: the two variants have the same conversion rate
• H₁: the variants have different conversion rates

The asymmetry is the whole trick: you can never prove H₁ directly. You can only gather evidence that makes H₀ look ridiculous, and reject it. That is why statisticians say “failed to reject the null” rather than “proved the null” — absence of evidence is not evidence of absence. It is the courtroom stance: innocent until the data make innocence absurd.

Two ways to be wrong

Because the verdict is a decision under uncertainty, there are exactly two ways to blow it — and they are the same two errors a spam filter makes.

• Type I error (α) — a false positive: you claimed a difference that isn’t real. The “significance level,” usually 0.05, is the maximum Type I rate you agree to tolerate.
• Type II error (β) — a false negative: there was a real effect and you missed it. Its complement 1 − β is the statistical power of the test.

The two are tied on a seesaw. With a fixed sample size you cannot drive both to zero — lower α (demand stronger evidence) and β rises (you miss more real effects). The α = 0.05 line is a convention, not a law of nature. Move the threshold below and feel the trade in your hands.

The p-value, said carefully

The previous lesson dared you with a question — when someone says “95% confident,” what is the 95% a probability about? Here is its twin, and the same trap. The p-value is the probability of seeing data at least as extreme as yours, if H₀ were true. A tiny p-value means “this result would be startling in a world where nothing is happening” — which is evidence against that world.

The misconception that sinks careers: a p-value is not P(H₀ is true). It is not a probability about the hypothesis at all — it is a probability about the data, computed assuming H₀. Just as a 95% interval’s randomness lives in the interval and not in the fixed truth, a p-value’s probability lives in the data and not in the hypothesis. Calling p = 0.03 “a 97% chance the effect is real” is the single most common misreading in all of published research.

It is also not any of these:

• the probability that your finding is real,
• the size or importance of the effect,
• a measure of how much you should trust the result.

A p = 0.04 from 100 samples with a microscopic effect is a far weaker finding than p = 0.04 from 100,000 samples with a large one. The p-value is decision input number one — never the whole story.

Two-sample t-test: are these groups different?

The bread-and-butter A/B setup: two independent groups, and the question of whether their means differ. Here the true rates are 4.0% (control) and 4.3% (treatment) — a real but tiny 0.3-point edge — with 2000 users per arm.

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)

# Group A — control. True conversion 4.0%
group_a = rng.binomial(1, 0.040, size=2000)

# Group B — treatment. True conversion 4.3% — a small but real lift
group_b = rng.binomial(1, 0.043, size=2000)

print(f"A conversion: {group_a.mean():.4f}")
print(f"B conversion: {group_b.mean():.4f}")

# Welch's t-test (doesn't assume equal variances)
t, p = stats.ttest_ind(group_a, group_b, equal_var=False)
print(f"\nt-statistic: {t:.3f}")
print(f"p-value:     {p:.4f}")
print("Significant at α=0.05?" , p < 0.05)

A conversion: 0.0425
B conversion: 0.0400

t-statistic: 0.397
p-value:     0.6911
Significant at α=0.05? False

Read that carefully, because it is the most honest thing in this lesson. The true rate of B is higher — yet in this sample of 2000, B came out below A (0.0400 vs 0.0425). The test, quite correctly, sees nothing: p = 0.69, nowhere near significant. At n = 2000 the sampling noise (σ/√n) is simply larger than a 0.3-point signal, so the experiment cannot even recover the sign of the effect, let alone certify it. The fix is not a cleverer test — it is power. Push to 20,000 per arm and the noise shrinks by √10, the real lift surfaces, and the p-value falls. Sample size buys the ability to see small things.

(On reading the confidence interval that accompanies such a test: the true value is fixed and the interval is the random thing — the same trap we untangled in estimation & confidence intervals.)

One-sample t-test: did the mean move?

Use this when you have one group and a fixed reference value to beat — “is our average response time below 200 ms?”

import numpy as np
from scipy import stats

rng = np.random.default_rng(1)
# Latencies for the new model. We want to claim mean < 200ms.
# H₀: mean >= 200ms.  H₁: mean < 200ms  (one-sided, lower tail)
latencies = rng.normal(loc=195, scale=20, size=80)

t, p = stats.ttest_1samp(latencies, popmean=200, alternative='less')
print(f"Sample mean: {latencies.mean():.2f}ms")
print(f"t = {t:.3f},  one-sided p = {p:.4f}")
print("Reject H₀ (mean >= 200ms)?" , p < 0.05)

Sample mean: 193.57ms
t = -3.398,  one-sided p = 0.0005
Reject H₀ (mean >= 200ms)? True

Here the evidence is strong: a sample mean of 193.6 ms with n = 80 gives p = 0.0005, so we reject ”≥ 200 ms” comfortably. Pass alternative='less' (or 'greater') when the hypothesis has a direction; the default is two-sided. The cousin ttest_rel handles paired samples — the same users measured before and after — by testing the within-pair differences.

When you compare many things, p-values lie

Check 20 metrics at α = 0.05 and, by chance alone, about one will look “significant” even if nothing whatsoever is happening. This is the multiple-comparisons problem, and it is how teams fool themselves daily. The simplest guard is the Bonferroni correction: running k tests, only believe a result whose p-value clears the stricter bar α/k.

import numpy as np
from scipy import stats

rng = np.random.default_rng(42)

# Simulate 20 metrics where NOTHING is actually different
n_metrics = 20
p_values = []
for _ in range(n_metrics):
    a = rng.normal(0, 1, size=200)
    b = rng.normal(0, 1, size=200)   # same distribution
    _, p = stats.ttest_ind(a, b)
    p_values.append(p)

p_values = np.array(p_values)
alpha = 0.05
n_naively_significant = (p_values < alpha).sum()
n_bonferroni_significant = (p_values < alpha / n_metrics).sum()

print(f"'Significant' at p < 0.05:               {n_naively_significant}")
print(f"'Significant' after Bonferroni (α/20):   {n_bonferroni_significant}")
print(f"(Truth: 0 of them are real effects)")

'Significant' at p < 0.05:               1
'Significant' after Bonferroni (α/20):   0
(Truth: 0 of them are real effects)

Exactly as the arithmetic warned: one of twenty crosses the naive line despite zero real effects, and Bonferroni’s stricter 0.05/20 = 0.0025 bar correctly throws it back. For hundreds of tests there are gentler procedures (Benjamini–Hochberg controls the false-discovery rate), but Bonferroni is the easy first reach.

A quick decision guide

In one breath

Hypothesis testing turns “is this difference real?” into a decision with a controlled error rate. You assume a null H₀ (“no effect”) and look for evidence extreme enough to reject it — you never prove H₁, only fail to reject H₀. Two errors trade off on a seesaw: Type I (α, false positive — the significance level you tolerate) and Type II (β, false negative), with power = 1−β; shrinking one with fixed n grows the other. The p-value is P(data this extreme | H₀ true) — a statement about the data, not P(H₀ true) and not the effect size. The toolkit: ttest_1samp (one group vs a value), ttest_ind (two groups), ttest_rel (paired); and when you test many things, correct the threshold (Bonferroni α/k) or you will manufacture false winners. Underpowered tests can’t even recover a small effect’s sign — the cure is sample size, not a fancier test.

Practice

A question to carry forward

You now have the grammar of testing — H₀, α, p-value, power, the t-test family. But grammar is not a sentence. The checkout example that opened this lesson is still unfinished: two real conversion rates, a borderline-looking lift, and a pile of operational ways to fool yourself that no formula warns you about.

So here is the thread onward into the applied version. How do you run a conversion-rate test end to end — the two-proportion z-test built directly on the CLT’s null distribution; the sample-size calculation that tells you, before launch, how many users you need to detect a lift you’d care about; and the four traps — peeking at the p-value early, fishing across multiple metrics, novelty effects, and powering for the wrong outcome — that quietly turn rigorous-looking tests into noise generators?