Covariance, Correlation & Total Expectation

The last lesson left us wanting a single number for how much two variables move together. Picture it. Plot a class’s study hours against their exam marks and the dots tilt upward — more study, higher marks. Plot hours of television against the same marks and the dots tilt down. Sometimes the dots are just a shapeless cloud with no tilt at all. Covariance is the one number that captures that tilt, and correlation is the same number polished onto a clean [−1, 1] scale so that “strong” means the same thing whatever the units.

Covariance — co-movement about the means

The form you compute from is the mean of the product minus the product of the means:

The sign carries the story. Positive covariance: an above-average X tends to ride with an above-average Y. Negative: they pull in opposite directions. Zero: no linear co-movement. And a tidy special case — set Y = X and the formula becomes Cov(X, X) = E[X²] − (E[X])² = Var(X). Variance is just covariance with itself.

Correlation — the same number, rescaled

Covariance carries the units of X times Y, so its raw size is hard to read. Dividing by both standard deviations strips the units away and pins it down:

ρ(X, Y) = Cov(X, Y) / (σ_X · σ_Y),       with   −1 ≤ ρ ≤ 1

ρ = +1 is a perfect rising line, ρ = −1 a perfect falling one, ρ = 0 no linear relationship. The [−1, 1] bound is guaranteed — a correlation outside it is an arithmetic mistake.

Variance of a sum carries a covariance term

Variances do not simply add unless the cross-term vanishes:

Var(X + Y) = Var(X) + Var(Y) + 2·Cov(X, Y)

Only when Cov(X, Y) = 0 (in particular, when X and Y are independent) does this collapse to the familiar Var(X) + Var(Y).

A worked example — a real GATE DA 2024 question

Toss two fair coins. Let X = 1 if both are heads (else 0), and Y = 1 if at least one is heads (else 0). Find Cov(X, Y).

The four equally likely outcomes are HH, HT, TH, TT. Get the three pieces, then subtract:

E[X]  = P(both heads)        = 1/4
E[Y]  = P(at least one head) = 3/4
E[XY] = P(X=1 and Y=1)       = 1/4     (X=1 already forces Y=1, so XY=1 only on HH)

Cov(X, Y) = E[XY] − E[X]·E[Y]
          = 1/4 − (1/4)·(3/4)
          = 1/4 − 3/16
          = 1/16 = 0.0625

So Cov(X, Y) = 1/16 = 0.0625, a verified GATE DA 2024 answer. It is positive, as predicted: X = 1 guarantees Y = 1, so the two indicators rise together. For 0/1 indicators this is always the shortcut — E[XY] is just the chance that both indicators are 1.

A question to carry forward

Every mean and variance so far assumed we knew the whole population. But in practice you almost never do — you have a sample of a few hundred and must reason about the millions behind it. Here is the thread onward: if you average a sample, how close is that average to the true mean, and what shape does the averaging produce?

In one breath

• Covariance Cov(X,Y) = E[XY] − E[X]E[Y] = the tilt: positive (together), negative (opposite), zero (no linear co-movement); Cov(X,X) = Var(X).
• Correlation ρ = Cov/(σ_X·σ_Y) rescales it to [−1, 1] (outside is an arithmetic error).
• Variance of a sum: Var(X+Y) = Var(X) + Var(Y) + 2·Cov(X,Y) — variances add only when Cov = 0.
• Independent ⇒ Cov = 0, but Cov = 0 ⇏ independent (Y = X² on symmetric X: dependent yet uncorrelated).
• The reliable NAT: two 0/1 indicators, where E[XY] = P(both = 1). 2024: two coins → Cov = 1/16 = 0.0625.

Practice