Conformal prediction

At 4:57 p.m., a delivery model predicts that a parcel will arrive in 42 minutes. The dispatcher has to choose between promising “before 5:30” and sending the job to a human. The number 42 is precise. It is not necessarily useful.

A point prediction tells you the centre of the model’s guess. It does not tell you how often the guess misses by 2 minutes, 20 minutes, or an hour. A model can have an excellent average error and still produce a dangerous number for the particular order now sitting on the screen.

What the dispatcher needs is something like:

Estimated arrival: 42 minutes.
90% prediction interval: 31 to 55 minutes.

That interval is useful only if “90%” means something operational. Conformal prediction attaches that meaning without assuming that errors are Gaussian, that the model is correct, or that the world follows a particular likelihood.

It does require one important assumption. We will get to the catch. It is the catch.

The promise: count the model’s misses

Suppose a model predicts a numeric outcome. For a known example with actual value y and prediction f(x), define a nonconformity score, a number measuring how badly that example disagrees with the model.

For ordinary regression, the simplest score is the absolute residual:

score = |y - f(x)|

Small scores look normal. Large scores look unusual.

Conformal prediction uses a separate calibration set, data not used to fit the model, to collect these scores. It asks how large a score must be so that only an alpha fraction of calibration examples are worse. If the answer is q, a new prediction becomes:

[f(x) - q, f(x) + q]

For the delivery model, a prediction of 42 minutes with q = 11 gives an interval of 31 to 53 minutes. The interval is not a claim that the model knows the true arrival time. It is a claim about the long-run frequency with which this construction contains the true arrival time.

The model can be a random forest, neural network, linear regression, or hand-written rule. Conformal prediction sits around the model and does not inspect its internals.

The assumption: exchangeability

Exchangeability means that the joint data would have the same probability distribution if you rearranged the examples. No example has a special position in the probability law.

Independent and identically distributed data, or iid data, is exchangeable. For ordinary tabular prediction, the practical recipe is simpler: randomly split data from a stable population, and ensure that calibration examples and future examples come from the same process.

Keep the fitted model fixed. Calculate scores for the n calibration examples, then calculate the score the future example would have if its outcome were revealed. Under exchangeability, the future score is equally entitled to occupy any of the n + 1 sorted positions. It is not systematically more likely to be the largest score than any calibration score.

Choose a threshold high enough that only an alpha fraction of those positions are above it. The future score then lands below the threshold with probability at least 1 - alpha. For regression, “score below the threshold” means the true y lies inside the interval:

P(Ynew is inside C(Xnew)) >= 1 - alpha

This is a finite-sample guarantee over a new example and the calibration sample. Ties make it conservative rather than invalid.

The model must be fitted without using calibration outcomes. Hyperparameter tuning, feature selection, or repeated inspection of calibration errors spends the calibration set; keep a fresh one. Full conformal prediction can use more data but may require refitting for candidate outcomes. Cross-conformal methods may improve data efficiency, but do not automatically have the same exact finite-sample guarantee as split or full conformal prediction.

Split conformal, one calculation at a time

Split conformal, also called inductive conformal prediction, gives the data three roles:

1. A training set fits the prediction model.
2. A calibration set measures its errors.
3. A future example receives an interval or set.

For regression:

1. Fit f on the training set.
2. For every calibration pair (xi, yi), calculate si = |yi - f(xi)|.
3. Sort the n scores.
4. Choose the required order statistic q.
5. For a new x, return [f(x) - q, f(x) + q].

A worked calibration set

Suppose the delivery model was fitted earlier. On ten calibration orders, its absolute errors in minutes were:

0.7, 1.1, 1.4, 1.8, 2.0, 2.2, 2.6, 3.0, 3.4, 4.1

We want 80% coverage, so alpha = 0.2 and n = 10.

Compute the rank:

k = ceil((n + 1)(1 - alpha)) = ceil(11 x 0.8) = ceil(8.8) = 9

The ninth-smallest score is 3.4 minutes. A new order predicted at 42 minutes receives:

[42 - 3.4, 42 + 3.4] = [38.6, 45.4]

Under exchangeability, at least 80% of future intervals made this way contain the actual arrival time. This does not mean there is an 80% chance that this particular finished interval contains its outcome. The interval is fixed once the input and calibration set are fixed; the probability describes repeated use of the procedure.

Why not use the ordinary 80th percentile of the ten scores? The future score is one of n + 1 exchangeable scores, not one of the calibration scores alone. The conformal quantile level is:

ceil((n + 1)(1 - alpha)) / n

The extra 1 reserves a possible rank for the future example. Dropping it can produce finite-sample undercoverage.

Small calibration sets make this discreteness visible. With n = 10 and desired 90% coverage:

ceil(11 x 0.9) = ceil(9.9) = 10

The threshold is the maximum calibration error, 4.1 minutes here. Thus the nominal 90% interval can in practice cover 100% of this calibration set. Ten observations cannot express every percentage smoothly.

For 95% coverage:

k = ceil(11 x 0.95) = ceil(10.45) = 11

There is no 11th calibration score. Do not silently replace it with the maximum: with distinct scores, the maximum gives only 10 / 11, about 90.9%, under the rank argument. A finite deterministic threshold cannot guarantee a target above n / (n + 1). When k > n, use q = +inf, producing an unbounded regression interval or an all-class classification set, or use a separately justified randomized method.

A runnable implementation looks like this:

import numpy as np

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(7)

# Synthetic delivery-like data: one feature, arrival time in minutes.
X = rng.uniform(0, 10, size=(300, 1))
y = (
    30
    + 3 * X[:, 0]
    + 2 * np.sin(X[:, 0])
    + rng.normal(0, 1, size=300)
)

X_train, X_rest, y_train, y_rest = train_test_split(
    X, y, test_size=0.40, random_state=1
)
X_cal, X_test, y_cal, y_test = train_test_split(
    X_rest, y_rest, test_size=0.50, random_state=1
)

model = RandomForestRegressor(
    n_estimators=200,
    random_state=1,
)
model.fit(X_train, y_train)

# Calibration scores: absolute errors on data not used for fitting.
calibration_predictions = model.predict(X_cal)
scores = np.abs(y_cal - calibration_predictions)

alpha = 0.10
target_coverage = 1 - alpha
if not 0 < target_coverage <= 1:
    raise ValueError("target coverage must be in (0, 1].")

n = len(scores)
if n == 0:
    raise ValueError("at least one calibration example is required.")

k = int(np.ceil((n + 1) * target_coverage))
if k > n:
    # No finite deterministic threshold can reach this target.
    q = np.inf
else:
    q = np.sort(scores)[k - 1]

point_prediction = model.predict(X_test[:1])[0]
interval = (point_prediction - q, point_prediction + q)

print(f"90% interval half-width: {q:.2f} minutes")
print(f"Prediction interval: {interval[0]:.2f} to {interval[1]:.2f} minutes")

The synthetic data make the code self-contained. The guarantee comes from the split, score calculation, and exchangeability—not from random forests.

Scores for classification

A classifier predicts class probabilities or scores. Conformal prediction turns them into a prediction set, which may contain one class or several.

For a classifier producing p(y | x), use the simple nonconformity score:

score(x, y) = 1 - p(y | x)

A confident correct prediction has a small score. A true class assigned probability 0.15 has score 0.85.

Calibrate these scores on held-out labelled examples. For a new input, include every class c whose score is at most the calibrated threshold:

C(x) = {c: 1 - p(c | x) <= q}

Equivalently, include classes whose predicted probability is at least 1 - q. A clear case might return {on time}; an ambiguous case might return {on time, delayed}. The guarantee concerns whether the true class is in the set, not the set’s size.

The probabilities do not need to be perfectly calibrated for the coverage proof: they rank candidate labels. Better probabilities often produce smaller sets, however. Conformal prediction guarantees coverage; it does not improve the model’s ranking. For probability calibration, see model calibration.

Marginal coverage is not individual coverage

The headline guarantee is marginal coverage:

P(Ynew is inside C(Xnew)) >= 1 - alpha

It averages over the population of future inputs. It does not promise 90% coverage for every possible input x.

Delivery times may be easy to predict in the city centre and chaotic during a snowstorm. An overall 90% interval could cover 98% of city-centre deliveries and only 50% of storm deliveries.

A stronger statement would be conditional coverage:

P(Ynew is inside C(Xnew) | Xnew = x) >= 1 - alpha

Distribution-free methods cannot generally provide this for every exact x with useful, finite-width intervals. There may be only one calibration example resembling that input—or none. Stronger guarantees require more assumptions or much wider sets.

Marginal coverage can also hide group failures. If 90% of the population belongs to group A and 10% to group B, overall 90% coverage could coexist with 99% for A and 9% for B.

Mondrian conformal prediction calibrates separately inside known groups. For regression, calculate a threshold qg for each group g and use that threshold at prediction time. Assuming exchangeability within the group, the guarantee becomes marginal coverage within each group. For classification, class-conditional conformal prediction uses class-specific calibration scores and thresholds.

The cost is data. A group with only 12 calibration examples has a coarse quantile, possibly its maximum error. A class with no calibration examples has no distribution-free class-specific estimate. More partitions mean less data per partition.

The production failure: distribution shift

The most dangerous conformal failure is quiet. Offline evaluation says 90% intervals cover 91% of a test set. Six weeks after launch, rolling coverage is 72%. The code still runs. The world changed.

Conformal prediction relies on calibration examples and future examples being exchangeable. A new region, pricing change, sensor replacement, pandemic, or seasonal regime can violate that assumption. If future errors are systematically larger than calibration errors, the old threshold is too small.

Monitor delayed coverage, interval width, prediction-set size, coverage by meaningful slices, and the distribution of nonconformity scores. A system that maintains 90% overall coverage by producing a 500-minute interval for every delivery is technically compliant and operationally useless.

Under shift, recent-window calibration, weighted or online conformal methods, and explicit shift modelling may help. None restores a distribution-free guarantee for an arbitrary new distribution without additional assumptions.

What to remember

• Conformal prediction measures held-out model errors and turns a high quantile into an interval or prediction set.
• Exchangeability makes the future score’s rank symmetric with calibration scores, yielding finite-sample marginal coverage of at least 1 - alpha.
• The rank is ceil((n + 1)(1 - alpha)); with small n, it can reach the maximum error. If it exceeds n, a finite deterministic threshold cannot meet the target.
• Marginal coverage is not conditional or per-group coverage. Leakage, too little calibration data, and distribution shift still break usefulness.