datarekha
Patterns June 10, 2026

Peeking is why your A/B test lies (and CUPED is the fix)

Watching a live experiment and stopping the moment it looks significant can push your false-positive rate from 5% to over 26%. The discipline — and the variance-reduction trick — that make online tests trustworthy.

6 min read · by datarekha · mlopsab-testingexperimentationstatistics

Deploying a new model gets it running in production. Proving it’s actually better is a separate, statistical job — and it’s where most teams quietly fool themselves. The culprit is peeking: watching a live experiment and stopping the moment the numbers look good. Modern dashboards make peeking effortless, and it silently wrecks your conclusions.

Why peeking inflates false positives

Each time you check a running test and consider stopping, you give random noise another chance to cross the significance line. The effect is large, not subtle. Evan Miller’s classic demonstration is the canonical reference: stopping a test as soon as it reaches significance (with a max of 150 observations) produces false positives 26.1% of the time, instead of the assumed 5%. You think you’re running at 5% error; you’re actually wrong one time in four.

There are two honest fixes. The simple one: fix the sample size or duration in advance and don’t stop early. The sophisticated one: use statistics designed for continuous monitoring. Johari, Pekelis, and Walsh’s “Always Valid Inference” shows standard p-values are unreliable under continuous monitoring and develops always-valid p-values that stay valid even when you peek — an approach later integrated into Optimizely’s Stats Engine so users can watch tests without inflating false positives.

CUPED: reach significance with less traffic

When you don’t have the traffic to wait, variance reduction helps. CUPED (Controlled-experiment Using Pre-Experiment Data) was introduced by Microsoft researchers Deng, Xu, Kohavi, and Walker in 2013, using pre-experiment data to reduce metric variance by about 50% on Bing’s experimentation system. It works by removing the predictable part of each user’s metric — their behavior before the test — so the noise that remains is smaller. Microsoft’s Experimentation Platform implements CUPED as an industry-standard technique, estimating a linear model on pre-experiment baseline data and subtracting the explainable variance. Half the variance means roughly half the sample for the same statistical power.

The throughline: offline accuracy tells you what to try; a trustworthy online test — fixed sample size, guardrails, and CUPED for efficiency — tells you what’s true.

Skip to content