What is the difference between the null and alternative hypothesis?

How to think about it

The null hypothesis, H₀, is the precise baseline claim you test against, usually that there is no effect or no difference. The alternative hypothesis, H₁, also written Hₐ, is the competing claim that an effect or difference exists, possibly in a specified direction.

The important point is that a test does not prove H₁ directly. It asks whether the data would be unusually far from what H₀ predicts.

Why the distinction matters

Suppose an online store tests a new checkout button. Version A is the existing button, and version B is the new one. The business wants to know whether B changes the conversion rate, meaning the fraction of visitors who complete a purchase.

Let pA be A’s population conversion rate and pB be B’s population conversion rate. A two-sided test could state:

• H₀: pB - pA = 0
• H₁: pB - pA ≠ 0

H₀ says that any difference seen in the sample is just sampling noise. H₁ says that the underlying conversion rates differ.

This setup gives the test a reference point. If H₀ supplies no difference, we can work out how much random variation to expect when two equally effective buttons are shown to different visitors. We then compare the observed difference with that reference distribution, which is the range of results we would expect from repeated samples if H₀ were true.

Without H₀, “the result looks large” has no statistical meaning. A difference of 0.5 percentage points might be impressive with very little natural variation, or ordinary noise when conversion rates are volatile.

The alternative tells us what counts as evidence against that baseline. It can be non-directional, as above, or directional:

• H₁: pB - pA > 0 means B improves conversion.
• H₁: pB - pA < 0 means B reduces conversion.
• H₁: pB - pA ≠ 0 means either an increase or a decrease matters.

The null is not “the truth,” and the alternative is not automatically “the interesting claim.” They are roles in a decision procedure. The null is the claim for which we calculate the expected behaviour of the data.

A concrete example

Imagine the experiment collects these results:

Version	Visitors	Purchases	Conversion rate
A	10,000	500	5.00%
B	10,000	550	5.50%

The observed difference is 0.50 percentage points, or 0.005 as a proportion. That looks promising, but the question is whether a difference this large could reasonably arise by chance if the true rates were equal.

Under H₀, we pool the purchases:

(500 + 550) / (10,000 + 10,000) = 0.0525

The estimated standard error, which measures the typical sampling variation of the difference, is approximately:

sqrt(0.0525 × 0.9475 × (1/10,000 + 1/10,000)) ≈ 0.00315

The corresponding standardized statistic is:

0.005 / 0.00315 ≈ 1.59

For a two-sided normal approximation, that gives a p-value of about 0.11. A p-value is the probability of seeing a result this far from the null expectation, or farther, assuming H₀ is true and the test assumptions hold.

If the team chose a 5% significance level before the experiment, this result does not cross the threshold. The correct conclusion is fail to reject H₀.

That does not mean the buttons are proven identical. The approximate 95% confidence interval for the difference is about -0.12 to +1.12 percentage points. The data are compatible with a small disadvantage, no meaningful change, or an improvement of roughly one percentage point. The experiment simply did not separate those possibilities clearly enough.

A strong candidate would not say, “B improved conversion, but it was not statistically significant.” That quietly treats the point estimate as established fact. A better statement is: “B had a sample uplift of 0.5 percentage points, but the uncertainty interval includes no effect, so this experiment does not provide conventional evidence of a real difference.”

What rejection does and does not mean

If the p-value had been 0.01, we would reject H₀ at the 5% level. That would mean the observed result would be unusual under the no-difference model.

It would not mean:

• H₀ has a 1% probability of being true.
• H₁ has been proven with 99% certainty.
• The effect is large enough to matter commercially.
• The same result will definitely appear in a new experiment.

This distinction is where many otherwise polished answers fail. A p-value is conditional on H₀. It is not the probability that H₀ is true.

There are two common decision errors:

• A Type I error is rejecting H₀ when H₀ is actually true, such as declaring B better when the difference was noise. The significance level, often written α, controls the long-run rate of this error under the test procedure.
• A Type II error is failing to reject H₀ when a real effect exists. The probability of avoiding that error is called power.

Power depends on sample size, noise, the effect size worth detecting, and the chosen significance level. In the checkout experiment, a 0.5 percentage-point improvement may be real but difficult to distinguish from noise with 10,000 visitors per arm. More traffic reduces sampling uncertainty, roughly in proportion to the square root of the sample size. It does not, however, fix biased assignment or a broken measurement system.

The directional nuance

A one-sided test is appropriate only when the direction was decided before looking at the results and the opposite direction genuinely would not count as evidence for the stated decision.

For example, if the team cares only whether B improves conversion, the formal hypotheses are usually:

• H₀: pB - pA ≤ 0
• H₁: pB - pA > 0

The equality case is the boundary used to calculate the test. Saying H₀ is simply pB = pA is a useful shorthand, but it hides this detail.

You cannot run a two-sided test, notice that B went up, and then switch to a one-sided test to obtain a smaller p-value. That choice spends more false-positive risk in one direction and must be justified in advance. If a decrease would also be important, use a two-sided alternative.

The same logic applies when “no effect” is not the real business question. Suppose the company would accept B only if it is no more than 0.2 percentage points worse than A. This is a non-inferiority question, meaning the goal is to show that a new option is not unacceptably worse.

With Δ = pB - pA, the hypotheses could be:

• H₀: Δ ≤ -0.002
• H₁: Δ > -0.002

Rejecting H₀ supports non-inferiority. A conventional test of equality cannot prove this. Similarly, an equivalence test defines a practically acceptable interval, such as -0.2 to +0.2 percentage points, and treats effects outside that interval as the null to be ruled out.

Practical significance and real-world validity

Statistical significance is not practical significance. With ten million users, an improvement of 0.03 percentage points might produce a tiny p-value while adding almost no revenue after engineering and maintenance costs. With 200 users, a genuinely valuable improvement may fail to reach that same threshold.

The confidence interval and the business threshold should be read together. Ask:

1. What effect would change the decision?
2. Does the interval rule out effects smaller than that?
3. Was the experiment designed with enough power to detect it?

Also check whether the experiment supports a causal claim. Random assignment, meaning visitors are assigned to A or B by chance, helps make the groups comparable and supports a causal interpretation. An observational test on existing customers may find an association, but a p-value alone does not remove confounding, where another factor influences both the treatment exposure and the outcome.

Finally, account for repeated questions. If a team tests 20 independent metrics and treats each p-value of 0.05 as a discovery, the approximate chance of at least one false positive when all null hypotheses are true is 1 - 0.95^20, or about 64%. Multiple testing procedures, a pre-specified primary metric, or a holdout group may be needed.

A failure mode interviewers like

The first symptom of optional stopping is often a result that crosses the 0.05 threshold on Tuesday, gets announced, and then slips back above 0.05 when more data arrives. Repeatedly checking the same experiment and stopping at a convenient moment changes the false-positive rate; the reported p-value no longer describes the procedure that was actually used.

Another common symptom is an implausibly narrow confidence interval when users generate many events. Counting 100 page views from one person as 100 independent observations understates uncertainty because those views are correlated. The unit of analysis should usually match the randomization unit, such as one user rather than every click.

What they’ll ask next

Is failing to reject H₀ the same as accepting H₀?

No. It means the data did not provide enough evidence against H₀ under the chosen test. A wide confidence interval may indicate low power, while a narrow interval around a practically irrelevant effect gives stronger evidence that any important effect is absent.

What exactly does a p-value mean?

It is the probability, assuming H₀ and the test assumptions are true, of obtaining the observed test statistic or one more extreme according to the pre-specified alternative. It is not the probability that H₀ is true and not the probability that the result will replicate.

When should I use a one-tailed test?

Use one when the direction is specified before seeing the data and only that direction can answer the decision question. If both improvement and harm matter, use a two-tailed test. Never choose one-tailed testing merely because the observed result points in a convenient direction.

Say this in the interview

“H₀ is the precise baseline claim, usually no difference, while H₁ is the competing effect or direction; we evaluate how surprising the data would be under H₀, and rejecting H₀ provides evidence for H₁ without proving it.”