How do you calculate sample size, statistical power, and minimum detectable effect for an A/B test?
Sample size, power, and MDE form a three-way trade-off: fix any two and the third is determined. You choose the MDE based on business materiality, then solve for the sample size that delivers 80–90 % power at alpha = 0.05.
How to think about it
The core formula
For a two-sample z-test on a proportion, the per-group sample size is:
n = 2 * (z_alpha/2 + z_beta)^2 * p(1-p) / delta^2Where p is the baseline conversion rate, delta is the MDE (absolute lift you want to detect), z_alpha/2 = 1.96 for alpha = 0.05 two-tailed, and z_beta = 0.84 for 80 % power (0.842) or 1.28 for 90 % power.
Worked example
Baseline conversion = 5 %, desired MDE = 0.5 pp (a 10 % relative lift), alpha = 0.05, power = 80 %.
n = 2 * (1.96 + 0.84)^2 * 0.05 * 0.95 / (0.005)^2
≈ 2 * 7.84 * 0.0475 / 0.000025
≈ 29,800 per group (~60k total)At 10,000 daily users split 50/50, that is roughly 6 days of runtime. If you can only run for 3 days, either accept lower power (riskier) or raise the MDE threshold (meaning you will only detect larger lifts).
How to set the MDE
The MDE is a business decision, not a statistical one. Ask: “What is the smallest lift on this metric that would justify the engineering cost, or that would change the launch decision?” If a 0.1 % conversion lift is indistinguishable from noise in the business context, do not power for it.
What power actually means
80 % power means that if the true effect equals the MDE, you will detect it 80 % of the time. The other 20 % of the time you will incorrectly conclude no effect (Type II error). High-stakes decisions (pricing, core ranking) warrant 90 % power.
Use a power calculator (scipy, statsmodels, or any A/B test calculator) to verify — the formula above assumes equal group sizes and a normal approximation, which holds well for proportions with n 30+.