Covariance, Correlation & Total Expectation

The last lesson left us wanting a single number for how much two variables move together. Picture it. Plot a class’s study hours against their exam marks and the dots tilt upward — more study, higher marks. Plot hours of television against the same marks and the dots tilt down. Sometimes the dots are just a shapeless cloud with no tilt at all. Covariance is the one number that captures that tilt, and correlation is the same number polished onto a clean [−1, 1] scale so that “strong” means the same thing whatever the units.

Covariance — co-movement about the means

Start with what covariance is trying to measure, before any formula. Take one point: ask how far its X sits above or below X’s own mean, ask the same of its Y, and multiply those two deviations. Both above their means, or both below, and the product is positive. One above and one below, and it is negative. Average that product across the whole population and you have a tilt score — that is covariance.

Algebra folds that average of deviations into a shorter form, and it is the short form you actually compute from: the mean of the product minus the product of the means.

The sign carries the story. Positive covariance: an above-average X tends to ride with an above-average Y. Negative: they pull in opposite directions. Zero: no linear co-movement. And a tidy special case — set Y = X and the formula becomes Cov(X, X) = E[X²] − (E[X])² = Var(X). Variance is just covariance with itself.

Correlation — the same number, rescaled

Covariance carries the units of X times Y, so its raw size is hard to read. Dividing by both standard deviations strips the units away and pins it down:

ρ(X, Y) = Cov(X, Y) / (σ_X · σ_Y),       with   −1 ≤ ρ ≤ 1

ρ = +1 is a perfect rising line, ρ = −1 a perfect falling one, ρ = 0 no linear relationship. The [−1, 1] bound is guaranteed — a correlation outside it is an arithmetic mistake.

Variance of a sum carries a covariance term

Variances do not simply add unless the cross-term vanishes:

Var(X + Y) = Var(X) + Var(Y) + 2·Cov(X, Y)

The extra term has a plain reading. Two quantities that tend to rise together make their sum swing wider than either one alone would suggest, and two that offset each other make the sum steadier than you would guess. 2·Cov(X, Y) is exactly that extra swing, added when the covariance is positive and subtracted when it is negative.

Only when Cov(X, Y) = 0 (in particular, when X and Y are independent) does this collapse to the familiar Var(X) + Var(Y).

A worked example — a real GATE DA 2024 question

Toss two fair coins. Let X = 1 if both are heads (else 0), and Y = 1 if at least one is heads (else 0). Find Cov(X, Y).

The four equally likely outcomes are HH, HT, TH, TT. Get the three pieces, then subtract:

E[X]  = P(both heads)        = 1/4
E[Y]  = P(at least one head) = 3/4
E[XY] = P(X=1 and Y=1)       = 1/4     (X=1 already forces Y=1, so XY=1 only on HH)

Cov(X, Y) = E[XY] − E[X]·E[Y]
          = 1/4 − (1/4)·(3/4)
          = 1/4 − 3/16
          = 1/16 = 0.0625

So Cov(X, Y) = 1/16 = 0.0625, a verified GATE DA 2024 answer. It is positive, as predicted: X = 1 guarantees Y = 1, so the two indicators rise together. For 0/1 indicators this is always the shortcut — E[XY] is just the chance that both indicators are 1.

A question to carry forward

Every mean and variance so far assumed we knew the whole population. But in practice you almost never do — you have a sample of a few hundred and must reason about the millions behind it. Here is the thread onward: if you average a sample, how close is that average to the true mean, and what shape does the averaging produce?

In one breath

• Covariance Cov(X,Y) = E[XY] − E[X]E[Y] = the tilt: positive (together), negative (opposite), zero (no linear co-movement); Cov(X,X) = Var(X).
• Correlation ρ = Cov/(σ_X·σ_Y) rescales it to [−1, 1] (outside is an arithmetic error).
• Variance of a sum: Var(X+Y) = Var(X) + Var(Y) + 2·Cov(X,Y) — variances add only when Cov = 0.
• Independent ⇒ Cov = 0, but Cov = 0 ⇏ independent (Y = X² on symmetric X: dependent yet uncorrelated).
• The reliable NAT: two 0/1 indicators, where E[XY] = P(both = 1). 2024: two coins → Cov = 1/16 = 0.0625.

Practice