Picking the right metric
Accuracy is a trap on imbalanced data. Here's the full crew — precision, recall, F1, ROC-AUC, PR-AUC, log-loss, MAE, RMSE, R² — and when to reach for which.
What you'll learn
- The confusion matrix as the foundation of every classification metric
- Precision, recall, F1, ROC-AUC, PR-AUC — and when each one lies
- Regression metrics — MAE, MSE, RMSE, R²
- Why metric choice is a business decision, not a technical one
Before you start
If you remember one thing from this lesson, make it this: accuracy is the worst metric in ML. Not because it’s broken — because it’s so intuitive that people use it on problems where it lies. A model that predicts “no fraud” on every transaction in a 99.7%-clean dataset gets 99.7% accuracy and is useless. Picking the right metric is a business decision, not a statistical one.
The confusion matrix is everything
Every classification metric can be derived from four numbers:
| Predicted: positive | Predicted: negative | |
|---|---|---|
| Actual: positive | True Positive (TP) | False Negative (FN) |
| Actual: negative | False Positive (FP) | True Negative (TN) |
In a churn model:
- TP — flagged a real churner. Win — you can intervene.
- FN — missed a churner. They leave silently. Expensive.
- FP — flagged a loyal customer. You send them a retention discount they didn’t need. Annoying but cheap.
- TN — correctly ignored a loyal customer. Status quo.
The cost of FN vs FP is what makes metric choice business-specific.
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
rng = np.random.default_rng(0)
n = 1000
X = rng.normal(size=(n, 4))
y = (X[:, 0] + 0.3 * rng.normal(size=n) > 1.5).astype(int) # ~7% positive
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
pipe = Pipeline([("s", StandardScaler()), ("m", LogisticRegression(class_weight="balanced"))])
pipe.fit(X_tr, y_tr)
preds = pipe.predict(X_te)
print("Confusion matrix [[TN, FP], [FN, TP]]:")
print(confusion_matrix(y_te, preds))
print()
print(classification_report(y_te, preds, target_names=["loyal", "churner"]))classification_report is the cheat sheet — it dumps precision, recall,
F1, and support for each class.
The four core classification metrics
Accuracy = (TP + TN) / total ← what fraction are correct
Precision = TP / (TP + FP) ← of those I flagged, how many were real
Recall = TP / (TP + FN) ← of the real positives, how many did I catch
F1 = 2 · (P · R) / (P + R) ← harmonic mean of P and RWhen to use which:
| Scenario | Metric | Why |
|---|---|---|
| Balanced classes, equal costs | Accuracy | Honest summary |
| FP is expensive (spam filter — don’t junk real mail) | Precision | Of flagged spam, how much really is spam |
| FN is expensive (cancer screening) | Recall | Of real cases, how many did we catch |
| Both matter, classes imbalanced | F1 | Penalizes any of P or R being low |
Notice accuracy doesn’t appear for imbalanced problems. Predict-all-negative on 5% positives gives 95% accuracy — and zero recall. The majority class dominates the numerator, so accuracy happily reports a high number while the model catches none of the cases you care about.
ROC-AUC vs PR-AUC
.predict_proba() gives you a probability per row, not just a label.
Every threshold you pick produces a different (precision, recall) pair.
ROC and PR curves sweep all thresholds:
- ROC curve plots TPR (recall) vs FPR. ROC-AUC is the area under it — interpretable as “the probability the model ranks a random positive higher than a random negative”. 0.5 = random, 1.0 = perfect.
- PR curve plots precision vs recall. PR-AUC is the area under it.
The catch: ROC-AUC looks great on imbalanced data even when the model is mediocre. FPR = FP/(FP+TN) — on a highly imbalanced dataset TN is enormous, so even a model that produces many false positives keeps FPR low, and ROC-AUC stays deceptively high. For imbalanced classification, prefer PR-AUC, which puts precision and recall front and center instead.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
rng = np.random.default_rng(0)
n = 2000
X = rng.normal(size=(n, 4))
y = (X[:, 0] + 0.4 * rng.normal(size=n) > 2.0).astype(int) # ~2% positive
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
pipe = Pipeline([("s", StandardScaler()), ("m", LogisticRegression())]).fit(X_tr, y_tr)
probs = pipe.predict_proba(X_te)[:, 1]
print("positive rate:", y_te.mean().round(3))
print("ROC-AUC:", roc_auc_score(y_te, probs).round(3))
print("PR-AUC (average precision):", average_precision_score(y_te, probs).round(3))For 2% positives, you’ll often see ROC-AUC of 0.90+ while PR-AUC is much lower. The ROC number is real, but PR-AUC is what reflects how the model actually performs at the operating points you care about.
Log-loss — for probability calibration
If you’re going to USE the probability (for expected-value calculations, bid pricing, threshold tuning), measure log-loss:
log-loss = −(1/n) Σ [ y log p̂ + (1−y) log(1−p̂) ]This is the same loss logistic regression trains on. It heavily punishes
confident wrong predictions. A model with great accuracy but lousy
log-loss has uncalibrated probabilities — its .predict_proba() outputs
aren’t trustworthy as actual probabilities.
Regression metrics
For continuous targets:
| Metric | Formula | When |
|---|---|---|
| MAE | mean(|y − ŷ|) | Robust to outliers, easy to explain to PMs |
| MSE | mean((y − ŷ)²) | What most models train on; heavily penalises big errors |
| RMSE | √MSE | Same units as target; the default “error in dollars” metric |
| R² | 1 − SS_res / SS_total | Unit-free, 1 = perfect, 0 = no better than predicting the mean |
A negative R² is not a bug — it means your model is worse than just predicting the average. That’s a meaningful diagnostic.
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(0)
n = 200
sqft = rng.uniform(600, 3000, n)
price = 0.2 * sqft + rng.normal(scale=40, size=n) # in $1000s
X = sqft.reshape(-1, 1)
X_tr, X_te, y_tr, y_te = train_test_split(X, price, test_size=0.3, random_state=0)
m = LinearRegression().fit(X_tr, y_tr)
pred = m.predict(X_te)
print(f"MAE: {mean_absolute_error(y_te, pred):>6.2f} (avg error in $1000s)")
print(f"MSE: {mean_squared_error(y_te, pred):>6.2f}")
print(f"RMSE: {mean_squared_error(y_te, pred) ** 0.5:>6.2f}")
print(f"R²: {r2_score(y_te, pred):>6.3f}")MAE vs RMSE is the classic regression-metric decision. RMSE is more sensitive to large errors (the residuals get squared), so a model that gets most rows right but blows up on a few outliers will have low MAE and high RMSE. Pick MAE when your business cares about typical error, RMSE when it cares about worst-case error.
The thresholding step
Most production classifiers don’t deploy at the default 0.5 threshold.
Once you have a calibrated .predict_proba(), you sweep the threshold
and pick the operating point that matches your business cost:
# Pseudo: for each threshold, compute expected business value
# (probs > t).sum() = number of rows flagged as positive at threshold t
for t in [0.5, 0.4, 0.3, 0.2, 0.1]:
flagged = (probs > t).sum()
true_pos = ((probs > t) & (y_true == 1)).sum()
ev = true_pos * value_per_TP - flagged * cost_per_FPThis step usually adds more value than another round of hyperparameter tuning. It’s also the step junior engineers most often skip.
How much do you trust the number?
Every metric above is a single number computed from one finite test set.
Swap in a different test split and you’d get a slightly different RMSE, a
slightly different F1. So how much does that number wobble? The
bootstrap answers this from the data you already have: resample your
test set with replacement, recompute the statistic, and repeat. The
spread of those resampled values is a confidence interval — no second
experiment required. Report metrics as 0.84 ± CI, not bare point
estimates, especially on small test sets.
In one breath
- Accuracy lies on imbalanced data — predict-all-majority scores high while catching none of the cases you care about; metric choice is a business decision.
- Every classification metric comes from the confusion matrix: precision = TP/(TP+FP) (cost of false positives), recall = TP/(TP+FN) (cost of false negatives), F1 = their harmonic mean.
- Sweep thresholds with curves: ROC-AUC (TPR vs FPR) flatters imbalanced data because TN is huge — prefer PR-AUC there; use log-loss when you’ll use the probability.
- Regression: MAE (typical error, outlier-robust), RMSE (worst-case, squares big errors), R² (unit-free; negative = worse than predicting the mean).
- Pick the metric before training (no metric-shopping); then tune the threshold to your cost ratio — usually more valuable than another round of hyperparameter tuning.
Quick check
Quick check
Questions about this lesson
What's the difference between precision and recall?
Precision is, of the items flagged positive, how many truly are; recall is, of all actual positives, how many the model caught. Precision punishes false alarms, recall punishes misses — and they usually trade off against each other.
When should I use F1 instead of accuracy?
Use F1 (the harmonic mean of precision and recall) when classes are imbalanced and you care about the positive class, because accuracy can look high while the model misses the rare cases. Accuracy is fine only when classes are balanced and errors cost the same.
What does AUC actually measure?
AUC (area under the ROC curve) is the probability that the model ranks a random positive above a random negative — a threshold-independent measure of how well it separates the classes. 0.5 is random; 1.0 is perfect ranking.
Practice this in an interview
All questionsA confusion matrix tallies predictions against ground truth in a 2x2 table: true positives, true negatives, false positives, and false negatives. From those four cells every classification metric — accuracy, precision, recall, F1, specificity — can be derived. It exposes *which kind* of error a model makes, not just how often it errs.
F1 is the harmonic mean of precision and recall: 2PR/(P+R). The harmonic mean penalises extreme imbalance between the two — a model with 1.0 precision and 0.01 recall gets F1 = 0.02, not 0.505. F1 is the wrong metric when the classes are heavily imbalanced or when the costs of false positives and false negatives differ sharply, in which case F-beta, PR-AUC, or a cost-weighted metric is more appropriate.
MAE, RMSE, MAPE, and R² each measure a different aspect of regression quality and each has a regime where it misleads. RMSE is dominated by outliers; MAE is robust but hides large-error tails; MAPE is undefined at zero and asymmetrically penalises under-prediction; R² can appear high even when absolute errors are large, and can be negative, yet is still commonly misread as a percentage-correct. Choosing the right metric requires knowing the cost structure of the prediction task.
The PR curve plots precision against recall as the decision threshold varies. On imbalanced datasets it is more informative than ROC because it ignores the large pool of true negatives that inflate ROC-AUC — a model that looks good on ROC can still have dismal precision, which PR-AUC immediately exposes. PR-AUC is the better metric whenever the positive class is rare and getting predictions right matters more than ranking.
Optimize precision when a false positive is costly — spam filters, ad targeting, legal evidence — because you'd rather miss some positives than act on wrong ones. Optimize recall when a false negative is costly — cancer screening, fraud detection, safety systems — because missing a true positive can be catastrophic. The business cost of each error type should drive the choice, not the metric itself.
RMSE (Root Mean Squared Error) penalises large errors quadratically, making it sensitive to outliers and appropriate when big deviations are disproportionately costly. MAE (Mean Absolute Error) treats all errors linearly, is more robust to outliers, and is easier to interpret in the units of the target. R-squared measures the proportion of target variance explained by the model — a value near 1 is desirable, but it can be high even for a bad model if the baseline variance is low, and it says nothing about prediction error magnitude.
The ROC curve plots True Positive Rate (recall) against False Positive Rate at every decision threshold. AUC — the area under that curve — equals the probability that the model ranks a randomly chosen positive example above a randomly chosen negative one. A random classifier scores 0.5; a perfect classifier scores 1.0.
The optimal threshold depends on the business cost of false positives versus false negatives, not on defaulting to 0.5. You choose it by plotting the PR or ROC curve on a held-out set, computing the metric that captures your cost function (e.g., F-beta, revenue, expected cost) at each threshold, and selecting the point that maximises it. Threshold tuning is free and should always precede resampling or model changes.