Covariance, Correlation & Total Expectation

Imagine plotting hours studied against exam score for a class. The dots tilt upward — more study, higher score. Imagine plotting hours of TV against the same score. The dots tilt down. Sometimes the dots are just a cloud with no tilt at all. Covariance is the single number that captures that tilt; correlation is the same number polished into a clean [−1, 1] scale so “strong” and “weak” mean the same thing across any pair. It is the same number a correlation heatmap reports when you screen features before fitting a model. GATE almost always tests these straight from the definition, often on a pair of 0/1 indicators (variables that are 1 when some event happens and 0 otherwise) you can enumerate by hand in a minute.

Covariance — co-movement around the means

The defining identity, the one form you compute from:

• Sign tells the story. Positive covariance: above-average X tends to pair with above-average Y. Negative: they move oppositely. Zero: no linear co-movement.
• Covariance with itself is the variance: Cov(X, X) = E[X²] − (E[X])² = Var(X). So variance is just the self-covariance — the same formula with Y = X.

Correlation — covariance, rescaled to [−1, 1]

Covariance carries the units of X times Y, so its raw size is hard to read. Dividing by both standard deviations strips the units and bounds it:

ρ(X, Y) = Cov(X, Y) / (σ_X · σ_Y),       with   −1 ≤ ρ ≤ 1.

ρ = +1 is a perfect increasing line, ρ = −1 a perfect decreasing line, ρ = 0 no linear relationship. The bound [−1, 1] is guaranteed — a correlation outside it is an arithmetic error.

Variance of a sum carries a covariance term

Variances do not simply add unless the cross-term vanishes:

Var(X + Y) = Var(X) + Var(Y) + 2·Cov(X, Y).

Only when Cov(X, Y) = 0 (in particular, when X and Y are independent) does this collapse to the familiar Var(X) + Var(Y).

How GATE asks this

A NAT that hands you a small experiment — frequently two 0/1 indicator variables — and asks for Cov(X, Y). The drill never changes: get E[X], E[Y], and E[XY], then subtract. Because the variables are indicators, XY = 1 only when both indicators are 1, so E[XY] is just the probability of that joint event — which is what makes these enumerable in seconds.

Worked example — a real GATE DA 2024 question

Toss two fair coins. Let X = 1 if both are heads (else 0), and Y = 1 if at least one is heads (else 0). Find Cov(X, Y).

The four equally likely outcomes are HH, HT, TH, TT, each with probability 1/4.

Step 1 — E[X]. X = 1 only on HH:

E[X] = P(both heads) = 1/4.

Step 2 — E[Y]. Y = 1 on HH, HT, TH (everything except TT):

E[Y] = P(at least one head) = 3/4.

Step 3 — E[XY]. XY = 1 requires X = 1 and Y = 1. But X = 1 (both heads) already forces Y = 1 (at least one head), so XY = 1 exactly on HH:

E[XY] = P(both heads) = 1/4.

Step 4 — apply the definition.

Cov(X, Y) = E[XY] − E[X]·E[Y]
          = 1/4 − (1/4)·(3/4)
          = 1/4 − 3/16
          = 4/16 − 3/16
          = 1/16
          = 0.0625.

So Cov(X, Y) = 1/16 = 0.0625 — this is a verified GATE DA 2024 question. The covariance is positive, which makes sense: X = 1 guarantees Y = 1, so the two indicators move together.

Quick check