Fairness & bias in ML

At 3 a.m., a lending team gets the question nobody wants to answer from a regulator:

“Your model is 92% accurate. Why are applicants in Group B approved at half the rate of applicants in Group A with similar financial profiles?”

The model may have passed its overall accuracy target. The dashboard may be green. The complaints are still real.

Fairness in machine learning is the discipline of checking whether a system’s errors and decisions fall unevenly across groups, then deciding which differences are acceptable and which are not. It is not a single score, and deleting a column called race or gender does not fix it.

Different ideas of fairness can disagree. Equal approval rates, equal error rates, and equally trustworthy scores are different goals.

Bias enters before the model sees the data

A model learns relationships between inputs and a target. If the target records past decisions instead of the outcome we care about, the model can reproduce an unfair process.

Suppose a bank trains on historical approval decisions. The target says whether an applicant was approved, not whether they would have repaid. If loan officers historically approved one neighbourhood less often, the model learns that pattern.

The algorithm did not invent the policy; it made the policy faster and harder to notice.

Common entry points include:

• Label bias: the target is a distorted measure of the desired outcome. An arrest record is not the same as committing a crime, and a hiring decision is not the same as future performance. Even repayment can reflect unequal loan terms.
• Sampling bias: some groups are scarce or absent in training data. A speech system trained mostly on one accent will make more errors on others.
• Measurement bias: the same underlying condition is measured differently across groups, such as symptoms recorded more often for patients who receive more attention.
• Proxy bias: features such as postcode, school, name, language, browser, or purchasing history encode protected attributes.
• Feedback loops: predictions change the future data. If a fraud system sends more transactions from one group for review, it generates more fraud labels for that group and may become even more suspicious.

Start with two questions: what outcome should the label represent, and who is missing from the data? Data collection, target definition, threshold, human review, and feedback all shape the result.

The mental model: scores become decisions

Most classifiers produce a score, often a probability-like number between zero and one. A threshold turns that score into an action: with a cutoff of 0.50, scores at or above it might mean “approve”.

A ground-truth label is the observed outcome used for evaluation. Let y=1 mean an applicant repaid a loan and ŷ=1 mean the model approved the application.

For each group:

• A true positive is a repaying applicant who is approved.
• A false negative is a repaying applicant who is denied.
• A false positive is a non-repaying applicant who is approved.
• A true negative is a non-repaying applicant who is denied.

“Qualified” or “positive” here means y=1 under the chosen label. It does not guarantee that the label is unbiased.

Two groups can have different score distributions, so one cutoff can produce different approval and error rates. Identical treatment is not necessarily equal impact.

A worked example with 200 loan applications

Consider 100 applicants in Group A and 100 in Group B. The label is whether the applicant repaid a comparable loan, and the approval decision is fixed.

Group A has 50 applicants who repaid and 50 who did not. It approves 40 of the repayers and 10 of the non-repayers.

Group B has 25 applicants who repaid and 75 who did not. It approves 15 of the repayers and 15 of the non-repayers.

import numpy as np

# y is the observed repayment outcome; y_hat is the approval decision.
y = np.array(
    [1] * 50 + [0] * 50 +   # Group A: 50 repaid, 50 did not
    [1] * 25 + [0] * 75     # Group B: 25 repaid, 75 did not
)

y_hat = np.array(
    [1] * 40 + [0] * 10 + [1] * 10 + [0] * 40 +  # Group A
    [1] * 15 + [0] * 10 + [1] * 15 + [0] * 60   # Group B
)

group = np.array(["A"] * 100 + ["B"] * 100)

for name in ["A", "B"]:
    rows = group == name
    repaid = rows & (y == 1)
    not_repaid = rows & (y == 0)

    selection_rate = y_hat[rows].mean()
    true_positive_rate = y_hat[repaid].mean()
    false_positive_rate = y_hat[not_repaid].mean()
    accuracy = (y_hat[rows] == y[rows]).mean()

    print(
        f"{name}: selection={selection_rate:.2f}, "
        f"TPR={true_positive_rate:.2f}, "
        f"FPR={false_positive_rate:.2f}, "
        f"accuracy={accuracy:.2f}"
    )

print(f"overall accuracy={(y_hat == y).mean():.3f}")

A: selection=0.50, TPR=0.80, FPR=0.20, accuracy=0.80
B: selection=0.30, TPR=0.60, FPR=0.20, accuracy=0.75
overall accuracy=0.775

Overall accuracy is 77.5%, but Group A’s selection rate is 0.50 and Group B’s is 0.30, a gap of 0.20.

Among applicants who repaid, approval is 0.80 for A and 0.60 for B, so a repaying applicant in B is more likely to be denied. The false-positive rate is 0.20 for both groups.

The average hides this pattern because it mixes groups.

Three definitions of fairness

Demographic parity, or statistical parity, compares selection rates:

P(ŷ=1 | group=A) = P(ŷ=1 | group=B)

It fails in the example: 0.50 ≠ 0.30. This criterion fits cases where equal access to a benefit is the main concern, but it does not ask whether selected people have the same outcomes.

Equalized odds compares both conditional error rates:

P(ŷ=1 | y=1, group=A) = P(ŷ=1 | y=1, group=B)

and

P(ŷ=1 | y=0, group=A) = P(ŷ=1 | y=0, group=B)

It fails here because TPR differs, even though FPR matches. Equal opportunity is the weaker version requiring only equal TPR.

Calibration asks whether a score means the same thing across groups. A score around 0.70 is calibrated if about 70% of people receiving that score have y=1, consistently across groups.

Calibration concerns scores; the other criteria concern decisions. See model calibration for its mechanics.

A base rate is the fraction of a group with y=1: 0.50 for A and 0.25 for B.

Suppose equalized odds uses TPR=0.80 and FPR=0.20. Expected selection is:

• Group A: 0.50 × 0.80 + 0.50 × 0.20 = 0.50
• Group B: 0.25 × 0.80 + 0.75 × 0.20 = 0.35

The error rates match, but selection rates do not, because the groups have different base rates. Conversely, enforcing parity at 0.50 while keeping B’s FPR at 0.20 would require:

0.25 × TPR + 0.75 × 0.20 = 0.50

That gives TPR = 1.40, which is impossible.

These are not universal leaderboard metrics. Choose the criterion according to the harm, the label, and the system’s purpose.

Reducing missed diagnoses may matter most in medicine; access, calibrated risk, affordability, and legal constraints may matter most in lending.

Mitigation: change the data, training, or decision rule

There are three main intervention points:

• Pre-processing: correct labels, collect missing examples, reweight observations, or remove features without a defensible relationship to the task. Reweighting gives existing examples more influence; it does not create new information.
• In-processing: add a disparity penalty or constraint to the training objective. This makes trade-offs explicit: reducing a TPR gap may lower accuracy, alter calibration, or increase another error.
• Post-processing: change thresholds or decisions after scoring. Equalized-odds methods may require randomized decisions when ordinary thresholds cannot satisfy the constraint. Group-specific thresholds can also create policy or legal concerns.

Tools such as Fairlearn can compute group metrics and compare mitigations. They cannot choose the right target or make a biased label unbiased.

Failure modes in production

The overall dashboard is green, but the group dashboard is red. Report counts and uncertainty for each important group and intersection, not just aggregate accuracy. A release should not pass by hiding a severe subgroup regression.

Removing the protected column does not remove the disparity. Proxies and interactions may still encode it. Compare score distributions and test intersectional slices such as age combined with disability status.

Offline fairness disappears after deployment. Production may have different base rates, missingness, or group proportions. Monitor selection rates and score distributions immediately; measure TPR and FPR once outcome labels arrive. These are delayed metrics, not optional ones.

You cannot evaluate denied applicants. This selective-label problem means repayment is observed mostly for approved applicants. Treating every denied applicant as a default quietly copies the old decision into the label.

Collect follow-up outcomes where appropriate or use causal analysis; see causal inference.

Small groups also produce noisy rates. Always show denominators, uncertainty intervals, and rolling windows before changing policy.

A production fairness process

Document the intended use, affected people, decision, label, and harm to prevent. Choose groups and metrics before inspecting results.

Report selection rate, TPR, FPR, precision, and calibration where relevant, including intersectional slices, geography, and time.

Use a deployment-representative holdout set and do not tune a threshold on the same data used for the final claim. Record the model version, data window, threshold, group definitions, sample counts, and mitigation choice.

Assign ownership for release decisions, complaints, and post-deployment review. Laws such as the EU AI Act impose obligations that depend on system, jurisdiction, and organisational role; they do not provide one universal fairness formula.

Fairness is a quality requirement with a target, measurement plan, trade-off, and owner.

Quick check

Next

Fairness is easier to audit when you understand what a model is doing. Continue with model interpretability, or inspect individual predictions with SHAP.