Simpson's paradox is already in your dashboard

Here is a result you can hand to a product manager and watch their week disappear.

A team ships a new checkout flow. Two weeks later the analyst pulls the numbers:

Segment	Old flow	New flow	Winner
Desktop	180 / 1,000 = 18.0%	40 / 200 = 20.0%	New
Mobile	20 / 500 = 4.0%	65 / 1,300 = 5.0%	New
All users	200 / 1,500 = 13.3%	105 / 1,500 = 7.0%	Old

The new flow wins on desktop. It wins on mobile. There are no other devices. And it loses overall by nearly half.

Nobody made an arithmetic error. Every cell in that table is correct, and you can add up the columns yourself to check. This is Simpson’s paradox, and once you can see the mechanism, you will start finding it in dashboards that nobody has ever flagged.

An aggregate is a weighted average, and the weights are not the same

The overall rate is not a fact about the flow. It is a fact about the flow and the mix of users who saw it.

Formally it is the law of total probability: the probability a user converts equals the probability they convert given desktop, times the share of users on desktop, plus the same product for mobile. The segment rates are one input. The segment shares are the other, and they carry equal weight.

In the table above, the old flow served two-thirds of its traffic to desktop, where everybody converts around four times better. The new flow served eighty-seven percent of its traffic to mobile, where nobody converts well. The new flow is being judged on a much harder population.

Reversal needs exactly two ingredients, and both are ordinary:

1. The segments have genuinely different base rates. Desktop converts at 18–20%, mobile at 4–5%. That gap exists in essentially every consumer funnel and has nothing to do with your experiment.
2. The two groups have different segment mixes. Old flow: 67% desktop. New flow: 13% desktop.

Neither ingredient is exotic. That is the uncomfortable part. Simpson’s paradox is not a rare statistical curiosity that shows up in textbook counter-examples; it is the default behaviour of pooled rates whenever assignment is correlated with a segment that affects the outcome.

Where the unequal mix comes from

In a properly randomised experiment with equal traffic allocation, this should not happen at scale — random assignment gives both arms roughly the same mix. So when you see a reversal, the first question is not statistical. It is: how did the assignment become correlated with the segment?

The usual answers are mundane and all worth checking:

• A staged rollout. The feature flag reached the mobile app a week before the desktop web release, or the other way around.
• A redirect or eligibility rule. Some users were routed away from the variant for a technical reason that correlates with device, region, or plan tier.
• A traffic-mix shift mid-window. A paid campaign, a seasonal spike, or a viral moment brought in a burst of users with a different profile, and the two arms did not accrue traffic at the same times.
• No randomisation at all. The “experiment” is a before/after comparison, or a self-selected opt-in, and the comparison groups differ in every way at once.

That last one is the most common by far, because most dashboard comparisons are not experiments. Comparing this quarter to last quarter, one sales region to another, or customers on plan A to customers on plan B is always a mix-confounded comparison.

Which number should you actually report?

Not automatically the segmented one. “Always disaggregate” is as wrong as “always pool,” and it fails in a specific way: if you keep slicing, you will eventually find a slice where the direction reverses purely by chance. Cut your data eight ways and one of them will look dramatic. That is a multiple-comparisons problem wearing a Simpson’s costume.

The honest procedure has three steps.

Decide the question before you look. “Which flow converts a given user better?” is a causal question about the flow, and the answer must hold the user population fixed. “How much revenue did we take last month?” is a descriptive question about what happened, and the pooled number is the right one — mix and all.

Pre-register your segments. Device, geography, new versus returning, plan tier — decide before the experiment which two or three splits you will examine, on the grounds that they have known base-rate differences. Anything you find outside that list is a hypothesis, not a finding.

Standardise before comparing. When the mix differs, reweight both groups to a common reference mix and compare the standardised rates. It is straightforward arithmetic. Take the combined population — 1,200 desktop sessions and 1,800 mobile across both arms — and apply each flow’s segment rates to that shared mix:

• Old flow: 0.18 × 1,200 + 0.04 × 1,800 = 216 + 72 = 288, so 9.6%.
• New flow: 0.20 × 1,200 + 0.05 × 1,800 = 240 + 90 = 330, so 11.0%.

Now the aggregate agrees with the segments: the new flow is better. This is direct standardisation, the same technique epidemiologists use to compare mortality between countries with different age structures, and it is not more complicated than that multiplication. Every analytics team should have it in a shared macro.

Catching it before it reaches a slide

Three practices, in increasing order of effort.

Report the mix next to the rate. Any dashboard tile showing a comparison of two rates should show the denominators broken out by the main segment. A reader who can see 1,000 versus 200 will ask the right question on their own. A reader who sees only 13.3% versus 7.0% cannot.

Alert on mix shift. You already alert when volume moves ten percent. Do the same for segment shares: mobile share, region share, new-user share, top-account share. A silent mix shift is the mechanism behind most metric mysteries, and the reversal case is only its most dramatic form.

Run the reversal check automatically. For any two-group comparison, it is a few lines of code to compute the pooled difference, compute the difference within each pre-registered segment, and flag when their signs disagree. Ship it as a warning on the dashboard rather than as a paragraph in a wiki nobody reads.

The part that generalises

Simpson’s paradox gets its attention because a full sign reversal is theatrical. But the reversal is only the extreme end of something that is happening constantly and quietly: every pooled metric you look at is partly a statement about your traffic mix.

Most of the time the mix moves a little and the metric moves a little, and nobody notices that half the movement was compositional. Overall margin falls while every product line’s margin rises, because the low-margin product grew. Average tenure drops while nobody left, because you hired. Average deal size falls while every segment’s deal size rose, because the self-serve tier launched.

The reversal is the case loud enough to catch. The attenuation is the case that has been quietly distorting your reporting all along, and the same fix handles both: never look at a rate without looking at what it is an average over.

Learn it as a system

Start with Averages that lie for the general failure of pooled summaries and the mix effects behind them, then read Conditional and total probability for the law of total probability that makes the reversal arithmetically inevitable rather than surprising. Finish with Cohort retention analysis, which is the standard production answer: fix the cohort, and the mix cannot move underneath you.