How do you design an A/B test from scratch?

How to think about it

I would design an A/B test as a pre-committed decision experiment: define the decision, hypothesis, metric, randomization unit, sample size, and end date before launch, then randomly assign eligible units to control and treatment. The goal is a credible estimate of what the change caused, large enough to matter and safe for important guardrails, not merely a difference that looks exciting in a dashboard.

Start with the decision and hypothesis

First, I name the decision the experiment must support. The control is the current experience. The treatment is the proposed change. Without a decision, teams often keep measuring until some metric looks favorable.

Suppose an online retailer wants to add price-drop badges to product cards. The decision is whether to show the badges to all eligible shoppers. My hypothesis would be: “Showing price-drop badges increases each user’s seven-day add-to-cart rate because the price information reduces purchase hesitation.”

The estimand is the exact effect I intend to estimate. Here, it is the difference in seven-day add-to-cart probability between users assigned to badges and users assigned to the current page. That wording matters. “Improve engagement” is not an estimand; it leaves the outcome, population, time window, and comparison vague.

I would pre-register the hypothesis, eligibility rules, primary metric, analysis method, sample size, end date, and planned exclusions. Pre-registration simply means writing the plan down before inspecting experiment outcomes. It prevents a very human failure mode: trying ten metrics, finding one positive result, and calling that result the hypothesis.

Choose the unit and make assignment random

Randomization means assigning each eligible unit to an arm by chance, so treatment assignment is not mixed up with user intent, geography, device type, or purchase history.

For this retailer, the unit should usually be a logged-in user. A user sees one assignment throughout the experiment, and the outcome is calculated once per user. Assigning pageviews would let one shopper see both versions, and assigning sessions could give the same shopper several supposedly independent observations. That makes the standard error look smaller than it really is.

A deterministic hash gives stable assignment without storing a separate random draw for every user:

import hashlib

def assign_variant(user_id, experiment_id):
    key = f"{experiment_id}:{user_id}".encode("utf-8")
    number = int(hashlib.sha256(key).hexdigest()[:8], 16)
    bucket = number / 2**32
    return "treatment" if bucket < 0.5 else "control"

The assignment key must include the experiment identifier, so two experiments do not accidentally share the same allocation. I would log the assignment event with the user ID, experiment version, timestamp, and eligibility decision.

The unit changes when users can affect one another. In a referral product, one user’s treatment may change what their friends see. That is interference, meaning one unit’s treatment affects another unit’s outcome. Individual randomization then violates a core assumption. I might randomize by household, team, geography, or time block instead, accepting that fewer independent units usually means lower statistical power.

Pick one primary metric and useful guardrails

The primary metric is the single outcome that determines the main ship-or-no-ship decision. For the badge test, I would use user-level seven-day add-to-cart rate: the fraction of randomized users who add at least one item to a cart within seven days of assignment.

I would also define guardrails: metrics that must not deteriorate materially. Examples are checkout completion, revenue per assigned user, page latency, refund rate, and support contacts. A badge could increase cart additions while attracting low-intent clicks and reducing completed purchases. That is not a successful experiment.

I would choose the metric before launch and specify whether “lift” means an absolute or relative change. Moving from 10% to 11% is a one-percentage-point absolute increase, but a 10% relative increase. Those are different claims.

Calculate sample size before looking at results

The calculation needs three main inputs:

• Baseline rate, the current expected value of the primary metric.
• Minimum detectable effect, or MDE, the smallest change worth acting on.
• Power, the chance of detecting the MDE if it is real, commonly 80% or 90%.

I would also set alpha, the pre-chosen false-positive tolerance. With a two-sided alpha of 0.05, the usual fixed-horizon test is designed to produce a false positive about 5% of the time when there is truly no effect, assuming the model and analysis plan are appropriate.

Suppose the retailer’s baseline add-to-cart rate is 10%. The business says a one-percentage-point absolute increase, from 10% to 11%, would justify the engineering and operational cost. With a two-sided alpha of 0.05 and 80% power, a standard two-proportion calculation needs roughly 14,700 users per arm, or about 29,400 users total.

That number is a traffic requirement, not automatically a runtime. If 20,000 eligible users arrive daily, the test can collect enough users in about a day and a half. I would not normally stop after a day and a half: weekday and weekend behavior differ, and the seven-day outcome for the final users has not matured. I would predefine a runtime covering complete weekly cycles, then wait seven days after the last assignment before calculating the final metric.

A 50/50 split gives the most information for a fixed total sample when the two arms have similar costs. If treatment is expensive to serve, an unequal split may be sensible, but it requires a new power calculation.

Validate instrumentation before trusting the result

Before launch, I would verify four links in the data chain:

1. The user was eligible.
2. The assignment was recorded.
3. The intended experience was actually rendered.
4. The outcome was recorded once and attributed to the correct user and time window.

An A/A test assigns users to two arms but gives both arms the same experience. It should not create a systematic difference. Running one is a useful way to find broken assignment, duplicated events, or a biased query before a real treatment is involved.

The first symptom of a serious assignment bug is often a sample ratio mismatch, meaning the observed arm counts do not resemble the planned allocation. A large experiment planned for 50/50 but showing 57/43 is a stop-and-investigate event, not a result to explain away after the fact. The cause might be client-side assignment failing on a device, an eligibility filter applied only to one arm, or users being counted differently in the dashboard and assignment table.

I would not repair that mismatch by deleting inconvenient users after seeing their outcomes. I would fix the pipeline, document the affected period, and rerun or reanalyze according to a pre-specified rule.

Analyze assigned users at the fixed horizon

The main analysis should use intention-to-treat, which means comparing users according to the arm they were assigned to, even if the badge failed to render for some of them. This estimates the effect of offering the feature in the real system, including imperfect adoption and delivery.

An exposure-only analysis can answer a different operational question, but it is often biased. Users who successfully load the badge may have faster devices, newer browsers, or higher intent. Those characteristics can affect purchasing independently of the badge.

Suppose the test ends with 50,000 users in each arm. The treatment has 5,500 add-to-cart users, or 11%. The control has 5,000, or 10%. The observed absolute difference is one percentage point, and the relative lift is 10%. Under a standard two-sample approximation, the 95% confidence interval for the absolute difference is roughly 0.62 to 1.38 percentage points.

A confidence interval gives a range of effects compatible with the data under the chosen statistical assumptions. If that interval excludes zero, the corresponding two-sided p-value is below 0.05 for the same model. A p-value is the probability of seeing a result at least this extreme if there were no true treatment effect; it is not the probability that the treatment is beneficial.

This result is statistically persuasive, but the business decision is still not automatic. The MDE was one percentage point, yet the interval includes effects below one point. I would combine the estimate and interval with revenue, guardrails, implementation cost, and the risk of long-term harm.

I would also keep the primary metric distinct from exploratory metrics. Looking for a win across 20 independent metrics at a 5% threshold produces at least one false positive about 64% of the time when all effects are actually zero. Multiple variants and repeated unplanned looks at the data create the same problem.

The senior-level nuance: when the textbook design breaks

An A/B test is strongest when units are independent, assignment is random, outcomes are measured consistently, and the effect appears within the chosen window.

That breaks in several common cases. Social features create interference. Marketplaces may need geography or time-block randomization. Fraud or safety experiments may have rare but severe outcomes that should not wait for an ordinary significance threshold. Retention changes may take months, so a seven-day win can be novelty rather than durable value.

When traffic is too low, randomization is impossible, or withholding treatment is unethical, I would not pretend an A/B test can answer the question cleanly. I might use a phased rollout, a geo experiment, a switchback design, or an observational causal method, while stating that those methods require stronger assumptions than random assignment.

What they’ll ask next

“Can you stop as soon as the p-value crosses 0.05?”
Not under an ordinary fixed-horizon design. Repeated peeking increases the false-positive rate. If the business needs continuous monitoring, I would use a sequential testing method with thresholds or an alpha-spending plan defined before launch.

“Why not randomize sessions?”
I would use sessions only when the effect is genuinely session-local and there is no carryover. For a shopping feature, users return, so user-level assignment avoids treating several decisions by the same person as independent evidence.

“What if add-to-cart is significant but revenue falls?”
The primary metric is not a permission slip. Revenue is a guardrail or possibly the better primary metric if it matches the decision. I would not ship a feature that wins on clicks by damaging completed purchases.

Say this in the interview

“I pre-register the decision and hypothesis, randomize the right unit, choose one business-relevant primary metric with guardrails, calculate the MDE and sample size before launch, validate assignment and logging, then analyze all assigned users at a fixed horizon and make the decision from both the confidence interval and the business impact.”