Bias & fairness in LLMs

A hospital gives its hiring assistant two identical résumés. One belongs to Aisha Khan; the other belongs to Alex Khan. The assistant writes “warm and supportive” for Aisha, “decisive and strategic” for Alex, then ranks Alex higher for a management role.

No single sentence looks like a smoking gun. The system may not even be making the final decision. Yet language has changed how two people are seen, and that language may influence who gets an interview.

That is the problem with fairness in large language models. A traditional classifier produces a label or score: approve the loan, flag the transaction, invite the candidate. An LLM also produces the explanation, summary, recommendation, refusal, and stereotype surrounding that decision. Sometimes the words are the harm. Sometimes they quietly steer the decision that follows.

Fairness in ML gives us the foundations for auditing decisions. LLMs require those foundations plus a careful account of what generated language does.

Two different harms

Allocational harm means distributing an opportunity, resource, or burden unfairly. A résumé screener that ranks one group lower is allocational. So is a support chatbot that gives one group worse medical guidance or an admissions assistant that recommends fewer applicants for human review.

Representational harm means depicting, describing, or treating a group in a stereotyped, demeaning, or erasing way. “The nurse… she” and “the CEO… he” are small examples. So are a résumé summary that calls a woman “pleasant” instead of “effective,” abusive language triggered by a dialect, or a system that never represents a minority group as an expert.

Here the output itself is the problem; there may be no binary decision or ground-truth label to compare. The hiring assistant has both kinds of risk. Its adjective choices create representational harm, while its ranking creates allocational harm. Fixing one does not fix the other.

A language model is not automatically a classifier because someone wraps its answer in a decision workflow. The model may generate text, a parser may extract a score, and a human may act on it. Audit every link in that chain.

What the model is actually doing

An autoregressive LLM predicts the next token—a small piece of text—from the tokens before it. For a prompt such as:

The nurse walked into the room. The nurse said that

the model assigns probabilities to possible next tokens. In a toy version, it might assign she a probability of 0.62, he 0.18, and other continuations the remaining 0.20.

It has not necessarily learned a sentence saying “nurses are women.” The association is distributed across patterns involving occupations, pronouns, names, dialects, and the contexts in which they appeared in training text. The practical result can still be a gendered continuation.

Repeat the prompt 1,000 times with fixed decoding. Suppose 880 completions use a female-coded pronoun and 120 use a male-coded pronoun. A simple signed skew is:

(male - female) / (male + female)

That gives (120 - 880) / 1,000 = -0.76. Zero means equal counts in this narrow test; a negative value means more female-coded completions.

The number is descriptive, not causal. It reflects one prompt and decoding configuration, not necessarily the training corpus, tokenizer, post-training, or prompt alone.

Why can a model amplify a corpus pattern?

• Next-token training rewards likely continuations, not balanced or socially desirable ones. Repeated stereotypes can therefore be useful to the objective.
• The relevant examples may be concentrated in particular genres, countries, and historical periods rather than representing a clean population sample.
• Context changes the distribution: “head nurse,” “ICU nurse,” and “nurse executive” can activate different associations.
• Sampling converts probability differences into observed frequencies. Temperature changes how often alternatives appear; it does not provide a theory of fairness.

Post-training adds another layer. RLHF, or reinforcement learning from human feedback, trains toward preferences collected from people comparing answers. It can reduce obvious toxicity while encoding raters’ assumptions about politeness, competence, politics, gender, or what a “helpful” person sounds like. A model that agrees too readily with a user—sycophancy—may reflect this preference-model problem. The alignment stage can reduce one bias while introducing or amplifying another.

Deployment supplies more conditioning. A system prompt may request “professional” writing; retrieval may contain historical records with unequal coverage; users may phrase equivalent requests differently. The output can then become a database label, training example, or evidence a reviewer trusts.

Bias is a pipeline, not a stain in one layer:

data → model associations → post-training preferences → prompt and context → output → human or system action

Changing one box can leave the others untouched.

Measuring a generation instead of guessing

Start by specifying:

1. The task: complete a sentence, summarize a résumé, answer a question, rank candidates, or refuse a request.
2. The slice: gender, race, disability, age, dialect, nationality, religion, or an intersection.
3. The comparison: matched prompts differing in one controlled feature where possible.
4. The metric: stereotype rate, quality, toxicity, factuality, refusal, or downstream decision. “Bias” alone is not a metric.

The occupation example is a generation-skew test:

# Illustrative counts from repeated generations.
outputs = {
    "nurse":    (12, 88),  # male%, female%
    "engineer": (90, 10),
    "teacher":  (40, 60),
    "ceo":      (85, 15),
}

for job, (male, female) in outputs.items():
    skew = (male - female) / (male + female)
    if skew > 0.3:
        tag = "skew MALE"
    elif skew < -0.3:
        tag = "skew FEMALE"
    else:
        tag = "~balanced"
    print(
        f"{job:<9} male {male:>3}% female {female:>3}%  "
        f"skew={skew:+.2f}  {tag}"
    )

It prints:

nurse     male  12% female  88%  skew=-0.76  skew FEMALE
engineer  male  90% female  10%  skew=+0.80  skew MALE
teacher   male  40% female  60%  skew=-0.20  ~balanced
ceo       male  85% female  15%  skew=+0.70  skew MALE

The threshold is an engineering choice, not a universal legal or moral boundary. The test makes a concern visible and repeatable, but a model could use gender-neutral language while assigning different competence, warmth, or danger to groups.

Counterfactual tests

For the hospital assistant, create matched résumé pairs. Keep education, experience, wording, and job history constant; change only the name or pronouns. Use the same instructions and decoding settings.

Compare more than the final rank:

• mentions of leadership, technical skill, warmth, or risk;
• positive versus tentative language and omitted achievements;
• recommendation and confidence;
• the reviewer’s decision after reading each summary.

If 100 identical pairs produce an interview recommendation for Alex in 78 cases and Aisha in 64, the counterfactual gap is 14 percentage points. That is an investigation signal, not proof of unfairness: names may carry cultural signals, formatting may differ, and the synthetic résumé may omit information used in real decisions.

Record the model version, system instructions, retrieval context, decoding settings, locale, and random seed where supported. Otherwise a later “fix” may simply be a different experiment.

Decision metrics still matter

Suppose the assistant recommends 48 of 100 qualified candidates in group A and 36 of 100 in group B. The true-positive rates are 48% and 36%; the selection-rate ratio is 36 / 48 = 0.75, with a 12-point absolute gap.

This measures allocational harm. Demographic parity compares selection rates without conditioning on qualification. Equalized odds compares true-positive and false-positive rates among people with the same ground-truth outcome. Calibration asks whether a score has the same meaning across groups—for example, whether a predicted 70% chance of success corresponds to roughly the same observed rate.

These criteria answer different questions and can conflict, especially with different base rates. Their labels may also encode past discrimination: “qualified,” “successful,” and “good cultural fit” are not objective merely because they appear in a spreadsheet.

Look beyond averages

Measure distributions, severe-output rates, worst-performing slices, and confidence intervals, not just means. An equal average sentiment score can hide degrading outliers. Also test quality parity: equal refusal rates do not help if one group receives less accurate answers. For multilingual or dialect-sensitive systems, compare factuality and useful completion, not only toxicity.

What the benchmark names mean

Each benchmark is a narrow probe:

• WinoBias and WinoGender test gendered coreference.
• BBQ tests whether a model relies on stereotypes when a question is ambiguous and follows evidence when context disambiguates it.
• StereoSet and CrowS-Pairs compare stereotypical and anti-stereotypical sentence preferences.
• BOLD evaluates open-ended generation across demographic prompts, including sentiment and toxicity.
• WEAT, the Word Embedding Association Test, measures associations in embedding spaces.

These are instruments, not report cards. They cover limited languages, identities, and formats; public tests may also be familiar to models through training or leakage. A strong score means only that the test found little problem under its conditions.

A production audit that has a chance

For the hospital assistant:

1. Collect representative, consented, or carefully synthetic cases, including intersections rather than one demographic column at a time.
2. Define harm with affected people. Missing achievements, demeaning descriptions, privacy, and lost opportunity may matter more than pronoun counts.
3. Run matched counterfactual and naturalistic cases. The first isolates a feature; the second reveals interactions clean tests miss.
4. Evaluate text and action. Store summaries, extracted scores, recommendations, overrides, and outcomes with appropriate access controls.
5. Have domain experts inspect examples behind the metrics, then repeat after model, prompt, retrieval, or workflow changes.

The first intervention is usually architectural. Keep the LLM as a drafting or evidence-organizing tool rather than the sole hiring decision-maker. Show the source résumé beside every extracted claim, require job-relevant justification for rejection, and log invented or omitted qualifications.

Blinding names can reduce direct identity effects but is not a guarantee. Schools, addresses, organizations, writing style, and employment history can act as proxies.

Mitigations have different failure modes:

• Data curation can remove slurs and improve coverage, but can erase legitimate discussion or balance superficial counts without changing context.
• Post-training and preference optimization can teach respectful responses while producing a polished version of the same bias.
• System instructions can request evidence-based, inclusive behavior but are brittle under unusual or adversarial prompts.
• Output classifiers and guardrails can catch slurs, while missing subtle stereotypes or blocking benign identity-related questions.
• Human review adds accountability but costs time and can reproduce social bias without guidance, evidence, and an appeal path.

Failure modes you will see first

The benchmark passes, but users report bias. A screenshot may show a patronizing résumé summary, a dialect interpreted as low competence, or a refusal triggered by an identity term. Add product-specific and slice-specific tests; generic benchmarks may never exercise that path.

The average score looks fine while the tail is ugly. Mean toxicity or sentiment can hide severe outliers. Track severe-output rates and worst-group results, inspect examples, and set escalation thresholds.

The model becomes politely useless. A benign question about disability, religion, or gender receives a refusal or lecture. Measure refusal rates on matched benign and harmful prompts, use intent-specific policies, and review false positives.

Removing protected attributes changes nothing. Proxy signals remain in names, schools, addresses, dialect, and work history. Test counterfactual changes across those signals and restrict the model to job-relevant evidence. A blind input is not necessarily a blind system.

The system improves, then regresses. A generated résumé summary can become training data for a later model, strengthening the original pattern. Keep generated labels distinguishable from human observations, monitor text and outcome metrics, and retain an audit trail.

Quick check

Next

This extends ML fairness into generation. The RLHF that can both reduce and introduce bias is alignment, and the proxy-gaming behind sycophancy is reward hacking.