Peeking makes A/B tests lie; CUPED reduces the traffic cost

At 10:14 on Tuesday, your new search-ranking model looks excellent. The experiment has 18,000 account-level assignments. Revenue per eligible account is $4.32 in treatment and $4.08 in control. The dashboard flashes p = 0.047.

Someone posts “ship it” in Slack.

At 10:35, 600 more accounts arrive. The difference is still positive, but the p-value is now 0.081. Nothing in the model changed. The sample changed. Your team just happened to inspect the experiment at a flattering moment.

That is the dangerous part of a live A/B dashboard: it does not merely report evidence. If you stop when the evidence looks good, the dashboard becomes part of the experiment’s decision rule. The nominal five-percent false-positive rate no longer applies.

Why peeking inflates false positives

A false positive, also called a Type I error, is claiming that treatment changed the outcome when in reality it did not. If the null hypothesis says “the ranking model has no effect,” a test run once at a predeclared endpoint might reject that null when p < 0.05 about five percent of the time.

That five percent is not a promise attached to every number that appears on a dashboard. It is a property of a particular procedure: choose the endpoint first, collect the data, and analyze it once.

Every new look gives random noise another chance to cross the line. If ten checks were independent, the chance of at least one p-value falling below 0.05 would be 1 - 0.95^10 = 0.401, or 40.1 percent.

Real interim looks are correlated because they reuse earlier users, so that calculation is not your exact false-positive rate. It shows the mechanism.

One concrete demonstration makes the number less slippery. In Evan Miller’s simulation, two equal-sized arms both had a null conversion rate of 10 percent.

The simulation first looked after 10 observations per arm, then after every additional 10 observations per arm, stopping at the first p < 0.05 and capping the test at 150 observations per arm. It used a two-sided Fisher’s exact test.

Under that null, 26.1 percent of the simulated experiments declared significance. That is the result of that allocation, outcome rate, look schedule, test, and stopping rule—not a universal false-positive rate attached to 150 observations.

The test is selecting lucky paths through the noise. Under the null, the estimated treatment effect wanders above and below zero. Most paths never look compelling. A few wander far enough to produce p < 0.05.

If your rule is “stop on one of those paths,” you preferentially keep the paths that got lucky.

A conventional p-value is calibrated for a fixed analysis time. It is not automatically calibrated for “the first time the p-value becomes small.”

This is why the same result can be statistically honest at a predeclared endpoint and dishonest when reported at the moment it first looked exciting.

Looking is not itself forbidden. If you inspect the dashboard every hour but still analyze exactly the predeclared endpoint, the fixed-horizon calculation can remain valid.

The practical danger is that people almost never look neutrally. They stop early, extend a promising test, change the primary metric, or quietly decide that a bad week was “just noise.” The analysis rule moves after seeing the data.

Two honest ways to monitor a live test

Predeclare the endpoint

The simple solution is to choose the sample size before launch. Define the minimum detectable effect, or MDE, as the smallest change worth acting on. For the ranking experiment, that might be a $0.20 increase in revenue per eligible account.

Set the baseline metric, significance level, desired power, randomization unit, and required sample from those inputs. But a baseline mean alone cannot calculate a sample size for a continuous metric such as revenue. You also need the baseline variance, or an explicit distributional model.

The calculation also needs:

• the allocation ratio;
• whether the test is one-sided or two-sided;
• expected attrition or missing outcomes;
• any clustering or design-effect assumption;
• the exact analysis window and outcome-maturity rule.

“Revenue per account” could mean revenue collected in the first seven days after assignment, not whatever has arrived by the time someone opens the dashboard.

The 74,000-account figure is therefore a fictional illustrative endpoint, not a calculation reproducible from the numbers in this lesson. In a real experiment, if the documented inputs produce a requirement of 74,000 eligible accounts, analyze at 74,000.

Power is the probability that the test detects the MDE if that effect is really present. An 80 percent power target means that a real effect of the chosen size will be detected in roughly eight out of ten comparable experiments.

It does not mean an 80 percent chance that your particular result is true.

Then choose one endpoint. Do not analyze at 18,000 because the p-value is attractive, then at 74,000 because the first result faded.

If traffic is unpredictable, predeclare a calendar endpoint or a sample endpoint and understand how that choice affects power. A maximum duration is useful operationally, but it does not magically turn an underpowered test into a conclusive one.

Use sequential inference

The more flexible solution is sequential inference, which is designed for repeated looks. Planned interim analyses can use an alpha-spending rule: each look receives only part of the total false-positive budget.

Always-valid p-values go further, but they are not ordinary p-values recalculated more often. An anytime-valid p-value is built from a sequential process, commonly an e-process.

An e-process is a nonnegative evidence process whose conditional expected value does not grow under the null. This is a nonnegative supermartingale.

Ville’s inequality says that if the process starts at one, the chance that it ever reaches 1 / alpha under the null is at most alpha. Converting that running crossing rule into a p-value gives a guarantee that survives continuous monitoring and arbitrary stopping times.

A fixed-horizon p-value has no such “ever” guarantee. Recomputing it after every new batch does not turn it into an anytime-valid p-value.

The following must also stay fixed:

• the metric;
• the target population;
• the estimand;
• the assignment procedure;
• the outcome window;
• the data-generating assumptions.

Changing the primary metric, eligible population, or definition of a mature outcome after seeing the data is still a new analysis decision.

Johari, Pekelis, and Walsh explain the distinction in “Always Valid Inference”.

This is the right answer when waiting for a fixed endpoint is genuinely expensive. It is not a license to copy a p-value formula from a blog post and call it sequential statistics.

Use a tested implementation, document the stopping rule, and test the implementation on A/A experiments. If the team cannot explain how its sequential method controls error, the boring fixed endpoint is safer.

CUPED: less noise, not a loophole

Once the stopping rule is honest, reduce the amount of traffic you need. This is where CUPED, or Controlled-experiment Using Pre-Experiment Data, earns its keep.

Adjust a pre-experiment metric

Suppose Y is an account’s outcome during the experiment: current-period revenue, for example. Let X be a related measurement from before randomization, such as the account’s revenue during the previous 28 days.

CUPED uses the pre-experiment value to remove the predictable part of the post-experiment outcome.

The basic adjusted metric is:

Y_adj = Y - theta_hat × (X - mean(X))

Here, theta_hat is the estimated coefficient connecting the pre-period measurement to the experiment outcome. The basic estimator is:

theta_hat = Cov(X, Y) / Var(X)

Cov(X, Y) is the sample covariance, which measures whether accounts high on X also tend to be high on Y. Var(X) is the sample variance of the baseline measurement.

The subtraction is centered around the average pre-period value so the adjusted metric stays on roughly the same overall scale.

Estimate theta_hat using one predeclared source: historical data containing comparable pre-period and outcome measurements, a treatment-blind experiment sample, or pooled experiment data under a fixed rule.

Do not estimate separate coefficients for treatment and control after seeing which arm appears to win.

With an independently estimated historical coefficient, experiment inference can generally treat the coefficient as fixed. If the coefficient is estimated from the same experiment outcomes, the standard error or randomization-inference procedure must account for that estimation.

A predeclared sample split is another way to separate coefficient estimation from effect estimation.

Also predeclare the rule for missing baselines. One option is to include every randomized account and impute missing X with a fixed training-set value, such as the overall pre-period mean.

Another is to analyze only accounts with a baseline observed for both arms. Either can be defensible when chosen in advance and applied identically. Switching rules after seeing the result changes the analysis population.

Take an account with $20 of prior-period revenue when the experiment-wide mean is $8. If theta_hat = 0.20 and the account produces $30 during the experiment, its adjusted outcome is:

$30 - 0.20 × ($20 - $8) = $27.60

An account with no prior revenue and no current revenue becomes:

$0 - 0.20 × ($0 - $8) = $1.60

That second number can look strange. It is an analysis value, not a new invoice.

CUPED is trying to compare treatment and control after removing predictable baseline differences. It is not claiming that every individual adjusted outcome is a physically possible amount.

Understand the traffic trade-off

The reason this works is variance. If X predicts Y, accounts with unusually high and low outcomes are less surprising after their history is considered.

The treatment effect is the difference between group means; randomization keeps the expected difference intact while the adjustment can make each group’s estimate less noisy.

Under the ideal linear model, the remaining variance is approximately the original variance multiplied by 1 - rho^2, where rho is the correlation between X and Y. If rho = 0.70, then 1 - rho^2 = 0.51: about 49 percent of the variance is removed.

This is variance, not standard deviation. Standard error behaves roughly like SE ~= sqrt(variance / n).

Halving the variance therefore cuts the required sample by about half for the same standard error, all else equal. The Bing experimentation work that introduced CUPED reported variance reductions around 50 percent for some metrics, and Microsoft describes the technique in its experimentation platform documentation. That is a useful result, not a universal guarantee.

CUPED helps only when the baseline measurement is available and predictive. For a brand-new account with no history, there may be little to adjust. If the correlation is near zero, the traffic savings are near zero.

An incomplete baseline join reduces coverage. Under a complete-case rule, fewer accounts remain usable and power can fall.

Under a fixed imputation rule, the affected rows may receive little or no variance reduction, so the predictive gain falls. A delayed join of a genuinely pre-assignment value is acceptable once the data arrive; it may postpone the final analysis, but the delay itself does not bias the treatment effect.

A post-assignment field is different and must be excluded. Treatment may have changed that field, so conditioning on it can remove part of the treatment effect or create collider bias.

Collider bias is where conditioning on a variable influenced by treatment and other causes of the outcome creates a spurious association. That is a causal-validity problem, not merely a complexity cost.

Use one predeclared adjustment rule for both arms. Estimate the relationship from the predeclared historical, pooled, or otherwise treatment-blind source.

Record the baseline coverage and estimated coefficient, and do not tune separate adjustments after discovering which arm appears to win. A more elaborate model may be useful, but it is another modeling decision that needs validation.

A statistical winner can still be a bad release

The ranking experiment’s primary metric might improve while the product gets worse. A model that increases checkout revenue by two percent but turns p99 latency from 180 milliseconds to 420 milliseconds is not a winner.

The p99 is the boundary below which 99 percent of requests fall; the slowest one percent still matters when those requests belong to your most valuable customers.

That is why every experiment needs guardrail metrics, which are safety measures that can block a launch even when the primary metric improves. Typical guardrails include:

• error rate;
• latency;
• cancellation;
• support contacts;
• revenue per account when the primary metric is a narrower engagement measure.

Check the assignment itself too. A 50/50 experiment with 20,000 eligible accounts should be near 10,000 accounts per arm.

A result such as 11,000 versus 9,000 is a sample ratio mismatch, or SRM: the observed allocation differs enough from the intended allocation to suggest a routing, eligibility, identity, or logging problem.

Trust-related guardrail metrics deserve a place on the main dashboard, not an appendix.

“No statistically significant latency harm” is also a weak guardrail. A low-powered test can fail to detect a harmful increase.

Set a practical harm threshold, such as “p99 must not increase by more than 20 milliseconds,” and inspect the confidence interval, which is a range of effects compatible with the data, against that threshold.

One production failure announces itself before anyone reads the final report: the adjusted result moves dramatically while the raw result does not, and baseline coverage differs between arms.

That often means the CUPED join is broken, the pre-period window is not actually pre-treatment, or one identity type is missing history. Stop interpreting the lift. Check the data pipeline.

The strongest objection

The fair objection is that fixed-horizon testing can waste time. If the treatment is clearly beneficial after 10,000 users, why force the business to wait for 100,000?

Do not force it. Use a sequential method. Early stopping for a large positive effect, a large negative effect, or a safety incident is perfectly reasonable when the rule was designed in advance.

What is unreasonable is calling an unplanned stop “common sense” and then reporting the ordinary fixed-horizon p-value as if nobody looked.

There is also a useful distinction between a product rollback and an efficacy claim. If latency doubles, roll back immediately. That is an operational safety decision.

Do not later describe the same early stop as proof that the treatment reduced conversion unless the statistical procedure supports that claim.

CUPED has a similar trade-off. It is not worth adding a fragile historical-data join to a tiny experiment with no predictive baseline. Raw, trustworthy data beats sophisticated, contaminated data every time.

What to do Monday morning

Write an experiment contract before changing the model. For the ranking example, specify:

• the account as the randomization unit;
• one assignment per account;
• the primary metric;
• an MDE such as $0.20 per eligible account;
• alpha of 0.05;
• target power of 0.80;
• the exact analysis window;
• the sample endpoint.

Add latency and error-rate thresholds as guardrails.

Make the dashboard show assignment counts, data freshness, baseline coverage, raw and adjusted estimates, and confidence intervals.

Show the ordinary p-value only at the planned endpoint, or show an always-valid p-value if you have adopted a sequential method. A dashboard should make a test easier to watch, not easier to stop dishonestly.

Before launch, calculate the CUPED baseline coverage and correlation on historical data. Confirm that every baseline field is measured before assignment.

Log the join version, coefficient, number of missing baselines, and treatment/control coverage. If those numbers change halfway through the experiment, treat that as a pipeline change, not a footnote.

At the endpoint, report:

• the actual sample;
• the SRM result;
• the metric definition;
• the raw effect;
• the CUPED effect;
• the confidence interval;
• the guardrail outcomes;
• any deviations from the contract.

If the experiment stopped early, record whether it stopped for a predeclared statistical boundary, a safety issue, or a business decision. Those are different facts.

For the wider system, A/B testing and experimentation covers the mechanics, deployment strategies puts tests beside canaries and rollbacks, and ML observability covers the production signals that guardrails depend on.

Offline accuracy tells you what to try. A disciplined online experiment tells you what survived contact with users. CUPED helps you get there with less traffic. It cannot make a lucky peek honest.