You can't be fair three ways — and the EU AI Act clock is ticking

At 3:07 on a Tuesday afternoon, a lending team is preparing to release a credit model. Its overall accuracy is 92 percent. The dashboard is green. Someone asks the question that tends to arrive late: “What does fair mean here?”

The model approves 56 percent of applicants in Group A and 38 percent in Group B. That gap might reflect unequal repayment rates. It might reflect biased training data. It might reflect both. The number alone cannot tell you which story is true.

My hard-won view is simple: stop asking whether a model is fair as if fairness were a property like latency or memory usage. Ask which kind of error you are willing to distribute, and why. The mathematics does not give you one universally correct answer.

It gives you several boundaries. In the binary example, different base rates make demographic parity and equalized odds conflict whenever the classifier carries signal. A separate result concerns calibration of a risk score, or predictive parity for a binary output, versus balanced error rates.

That is not an excuse to shrug. It is a reason to make the choice explicit, test it in production, and preserve the evidence.

Three fairness questions

A base rate is the share of people in a group who actually have the outcome being predicted. In our loan example, suppose 60 percent of Group A repaid within twelve months, while 30 percent of Group B did. Those rates are different before the model makes a decision.

Demographic parity means approving the same proportion of people in each group. If the model approves 500 of every 1,000 applicants in Group A, it should approve 500 of every 1,000 in Group B.

This asks: “Are people selected at the same rate?”

That can be the right question when unequal selection is itself the harm. It can also hide important information.

If the underlying repayment rates differ because of historical access to credit, equal approval rates may not produce equal outcomes. If the difference reflects real, well-measured risk, forcing equal selection may create losses or push lenders toward arbitrary decisions.

Equalized odds means that error rates are equal across groups, conditional on the true outcome. The true-positive rate, or TPR, is the share of people who would repay who are approved. The false-positive rate, or FPR, is the share of people who would not repay who are nevertheless approved.

Equalized odds asks two questions:

• Do equally qualified applicants get approved at the same rate?
• Do equally unqualified applicants get rejected at the same rate?

This is often a strong choice when both missed opportunities and bad approvals matter. It requires reliable labels, though.

In ordinary lending data, repayment is usually observed only after an approved loan is made. For a rejected applicant, “would repay” is a counterfactual outcome: what would have happened if the loan had been offered. It is therefore not a directly observed database field.

The observed outcome is shaped by the previous credit system, loan terms, income shocks, and who was approved in the first place.

Calibration means that a predicted risk has the same meaning across groups. If a score of 0.80 means “an 80 percent chance of repayment,” then people receiving that score should repay at roughly the same rate in every group.

Here, calibration refers to a risk score, usually continuous or with many possible values, rather than merely an approve-or-reject output.

Calibration asks: “Can a decision-maker trust the score in the same way for everyone?”

That matters for pricing, limits, and human review. A calibrated score can still lead to unequal approval rates because the score distributions can differ.

A threshold is the score cutoff used to approve. If one group has more people above that cutoff, approval rates differ even when the score has the same meaning. Calibration does not promise equal selection. It promises comparable meaning.

One nearby property needs its own name. Predictive parity means that among positive predictions, the positive predictive value, or PPV, is the same across groups.

In this example, PPV is the share of approved people who repay. Some fairness literature calls this binary calibration; here, to keep the objects separate, we call it predictive parity. It does not by itself tell us whether an underlying continuous risk score is calibrated.

These are not competing names for the same thing. They protect against different failures.

The contradiction in actual numbers

Return to a synthetic example with 1,000 applicants in each group. We stipulate each person’s outcome so the arithmetic is visible. This table is not a claim that a lender observes these labels for rejected applicants.

Group	Stipulated base rate: would repay	Stipulated would not repay
A	60 percent, or 600 people	400 people
B	30 percent, or 300 people	700 people

Suppose the model has the same error rates in both groups. It correctly approves 80 percent of people who would repay, so its TPR is 0.80. It incorrectly approves 20 percent of people who would not repay, so its FPR is 0.20.

That satisfies equalized odds.

For any group, the approval rate is:

TPR × base rate + FPR × (1 − base rate)

For Group A:

0.80 × 0.60 + 0.20 × 0.40 = 0.56

The model approves 560 people.

For Group B:

0.80 × 0.30 + 0.20 × 0.70 = 0.38

The model approves 380 people.

The model has identical TPR and FPR, but demographic parity fails by 18 percentage points. That gap is not a bug in the arithmetic. It follows from the different base rates.

The relationship is even clearer if you subtract the two approval rates:

approval gap = (base rate gap) × (TPR − FPR)

Here, that is 0.30 × 0.60 = 0.18.

The gap disappears only if the base rates are equal or if TPR equals FPR. The latter describes an uninformative classifier: approval is unrelated to repayment.

You can achieve equal approval rates by throwing away the predictive signal. That is technically fair by one definition and useless as a credit model.

Now examine the binary decision’s predictive parity, not the calibration of an unseen continuous score. We have only an approve-or-reject output here, so treat approved and rejected as the complete binary score: approved is one score value and rejected is the other.

Among approved applicants in Group A, the share who repay is:

0.80 × 600 ÷ 560 = 0.857

So the approved group has an 85.7 percent repayment rate.

Among approved applicants in Group B:

0.80 × 300 ÷ 380 = 0.632

Only 63.2 percent repay.

This is predictive-parity failure. The rejected class shows the same problem. The negative predictive value, or NPV, is the share of rejected people who would not repay:

For Group A:

0.80 × 400 ÷ 440 = 0.727

For Group B:

0.80 × 700 ÷ 620 = 0.903

The negative predictive values are 0.727 for Group A and 0.903 for Group B.

If approve and reject really are the entire score, calibration in this coarse binary sense requires the outcome rate for both score values to match across groups. Here neither class has the same meaning: the approved class has different PPV, and the rejected class has different NPV.

If the real system emits a continuous risk score, these aggregate PPVs do not establish that the score is uncalibrated. A calibrated continuous score can still produce different PPV above one threshold because the groups can contain different mixes of scores inside that approved band.

To test continuous calibration, examine score bands — narrow ranges such as 0.75 to 0.85 — and compare predicted risk with observed repayment separately by group, including counts and uncertainty. This synthetic binary table has no score bands, so it cannot answer that question.

The separate impossibility results need careful labels. Kleinberg, Mullainathan, and Raghavan study calibration of a risk score alongside “balance” conditions: equal mean scores among people who repay and among people who do not.

For a binary approve-or-reject score, those conditional means reduce to TPR and FPR. For a continuous score, balance is not simply the same condition as equalized odds.

Chouldechova studies predictive parity and error-rate balance for a binary predictor. Under unequal base rates, positive mass in both outcome classes, and non-perfect, nondegenerate predictions, each line of work rules out its respective calibration-or-predictive-parity and error-balance combination.

Neither paper is the source of the demographic-parity calculation above. That conflict follows directly from the displayed selection-rate identity.

The assumptions matter. If base rates are equal, the base-rate-driven contradiction in the binary arithmetic disappears and compatible choices become possible.

A perfect predictor can satisfy calibration and equalized odds. With unequal base rates, it still selects different proportions and therefore does not satisfy all three goals.

A random or constant predictor can satisfy some equal-rate criteria by throwing away useful signal. Those exceptions are why the result is not a claim that every combination of metrics is mathematically impossible. They also do not make a useless model acceptable.

Better predictions do not remove the policy choice

The strongest objection is also partly correct: perhaps the model is simply poor. Improve the data, add useful features, remove proxy variables, and the fairness problem may shrink.

Do that work. Better measurement can reduce avoidable error. A causal analysis may reveal that a feature is carrying the effect of discriminatory treatment rather than legitimate repayment risk. Better loan terms may change the repayment process itself.

But better accuracy does not make unequal base rates vanish. A perfect model that knows exactly who will repay still approves different proportions if one group contains more people who will repay.

Accuracy changes how well you estimate the outcome. It does not decide whether equal selection, equal errors, or equal score meaning is the governing principle.

The same applies to removing protected attributes. If you delete race or sex from the input table, you have not deleted geography, school history, income volatility, names, language, or past decisions that may encode it.

More importantly, you have removed the information needed to measure group disparities. A model cannot be audited for a gap that the pipeline refuses to record.

Fairness is therefore a decision about consequences. A lender might prioritize equal false-negative rates because denying a creditworthy applicant is the central harm. Another lender might prioritize calibrated risk because the score determines interest rates and capital allocation.

A public benefits system might put more weight on equal access. None of these choices can be made by an optimizer without smuggling in a value judgment.

The production failure is usually boring

The theorem is dramatic. The incident is usually not.

Suppose a new loan model raises overall accuracy from 91 percent to 92 percent. During retraining, the data mix changes. Group B has fewer observed outcomes because many recent loans have not reached the twelve-month repayment window.

The training job silently drops those rows. The model looks better overall.

An audit report may say that Group B’s false-negative rate, or FNR, rose from 14 percent to 27 percent. FNR is the share of people who would repay but are rejected.

That figure is a population FNR only when the “would repay” outcome is observed independently of the approval decision, or when a documented rejection-inference model — an explicit model for the unobserved outcomes of rejected applicants — or a randomized audit or offer design supports the estimate.

Otherwise it may be a rate on whatever labeled subset survived the selection process, or an estimate whose assumptions are hidden. Neither dropping immature rows nor restricting the data to applications with known outcomes cures rejection bias.

The first symptom may be a rise in appeals or manual reviews, not a red fairness dashboard. The dashboard may display N/A because labels are delayed. Or it may show a reassuring overall average that has diluted a small group’s result.

Fairness checks belong in the same lifecycle as accuracy checks.

Build the evaluation record

Start with a decision contract: a short record of the decision, the predicted outcome, the affected people, the legitimate use of each feature, and the harm the team is trying to limit.

Write down the chosen fairness definition before looking at a dozen metrics. Metric shopping after the results arrive is just p-hacking with better typography.

Then create a frozen evaluation set with a clear observation window. For a synthetic exercise, that might mean using applications made between January and June whose twelve-month outcomes are now known.

In production lending, however, the rows with known twelve-month repayment outcomes are usually approved loans, not a random sample of all applicants. The observation window fixes outcome maturity; it does not fix rejection bias.

State the estimand — what population the metric claims to describe — before reporting it. Distinguish observed repayment among approved loans from an estimate of the all-applicant counterfactual outcome.

An all-applicant TPR or FNR requires an outcome observable independently of approval, an explicitly assumed rejection-inference model, or a randomized audit or offer design. Otherwise report the missing-label denominator, sensitivity analysis, or bounds instead of presenting the FNR as directly measured.

Record the data version, label definition, group definitions, exclusions, and threshold. Do not tune the threshold on the same set used to claim the final result.

Report overall and per-group:

• the estimand and the number and proportion of applicants whose outcome is missing
• base rate
• selection rate
• TPR and FPR
• positive predictive value
• calibration by score band
• sample counts and uncertainty intervals

Counts matter. If a group has only 20 positive examples, one additional missed repayment changes its measured TPR by five percentage points.

A hard five-point gate on that estimate pretends to have more certainty than the data contains. Use minimum sample rules, confidence intervals, or a review path for low-volume groups. A gate that fails randomly will eventually be bypassed.

Check intersectional groups too. Intersectional evaluation means examining combinations such as age and disability rather than only one attribute at a time.

A model can look acceptable for each broad group while failing badly for a smaller combination.

Finally, separate model quality from decision policy. If the score is useful but the chosen threshold creates an unacceptable disparity, you have several options:

• change the threshold
• use a different decision rule
• collect better labels
• alter the product
• add human review
• reject the use case

Post-processing can move a metric. It cannot make the underlying trade-off disappear.

What the EU AI Act changes

As of 28 August 2026, the EU AI Act’s main obligations apply to high-risk AI systems covered by Annex III, including relevant systems used for creditworthiness and certain employment or essential-service decisions.

The Act’s timetable gives safety components of products covered by the separate product-safety route a later date, 2 August 2027. Classification and obligations depend on the system and on whether your organization is acting as provider, deployer, importer, or another role.

The published implementation timeline is a starting point, not legal advice.

The important engineering point is that the Act does not prescribe “use demographic parity.” It requires a risk-management process, appropriate data governance, technical documentation, logging, transparency and instructions, human oversight, accuracy, robustness, cybersecurity, and quality management for applicable high-risk systems.

Deployers have their own duties, including use according to instructions and monitoring.

For the central creditworthiness example, an applicable deployer of a high-risk AI system classified under Annex III point 5(b) must also perform and document the Article 27 fundamental-rights impact assessment before putting the system into use, then update it when relevant changes occur.

This duty depends on the system’s classification, the organization’s deployer role, and the scope of Article 27. It is not a blanket requirement for every model used anywhere in lending.

The Article 27 assessment is distinct from any data-protection impact assessment, or DPIA, required under data-protection law. A DPIA examines risks to people from processing personal data. The Article 27 assessment examines the fundamental-rights effects of using the high-risk AI system.

They can share evidence, affected-group analysis, and mitigations, but one does not replace the other.

Data governance means controls over how data is collected, checked, documented, versioned, and used. For the loan model, that includes why repayment is the label, how missing outcomes are handled, whether rejected applicants create selection bias, and which groups are represented in training and evaluation data.

A model card — a structured description of a model’s intended use, data, evaluation, limitations, and performance — is valuable evidence, but the Act does not prescribe a generic “bias-audit” or “model-card” format.

It is not a magic compliance document. A stale PDF cannot replace technical documentation, risk records, logs, human-oversight procedures, or a conformity assessment, a formal compliance check, where one is required.

The useful shift is to generate the model card from the same versioned artifacts that produced the model, then attach it to the release.

What to do on Monday morning

Confirm the legal role and scope first. If your organization is an applicable deployer of a high-risk creditworthiness system under Annex III point 5(b), perform and document the Article 27 fundamental-rights impact assessment before putting it into use, then update it when relevant changes occur.

Record the affected people and groups, plausible fundamental-rights harms, mitigations, and governance. Check whether a GDPR data-protection impact assessment is also required.

The DPIA concerns risks from processing personal data; the Article 27 assessment concerns fundamental-rights effects of using the AI system. Share evidence where sensible, but one does not replace the other.

Write the decision contract next. For the loan model, state whether the primary concern is equal access, equal error burden, or trustworthy risk estimates.

Get that choice signed by the product owner, risk owner, and the people responsible for compliance. Do not let the model’s first metric decide the policy by accident.

Create one reproducible evaluation command or job that emits the same report for every model version. Include subgroup counts, the base rate, selection rate, TPR, FPR, positive predictive value, score calibration, uncertainty, and the threshold.

Make missing labels visible instead of quietly dropping them.

Add a release rule with an owner and an exception path. “Group B TPR gap is 8 percentage points” is actionable. “Fairness failed” is not.

If the data is too sparse to judge, route the model to review rather than painting the result green.

Monitor the inputs and outcomes after release. Track group composition, score distributions, selection rates, and delayed TPR and FPR.

A fairness report that runs only at training time will miss the model that quietly rotted for six weeks after a policy change.

For the statistical background, see Fairness and bias in ML. For the operational pattern, Responsible-AI ops covers model cards, monitoring, and evidence.

The broader principle is the same as any serious production control: define the failure, measure it at the boundary, and make the system react before a person has to complain.

Fairness is not one number waiting to be maximized. It is a choice about whose error matters, made visible early enough that the choice can still be changed.