Bias & fairness in LLMs

Fairness in ML is usually framed around a classifier’s decisions — does the loan model approve groups at equal rates? LLMs inherit all of that and add a problem that’s specific to generating language: because they produce text, they can output, amplify, and normalize stereotypes and demeaning depictions — harm that a yes/no classifier’s fairness metrics don’t capture.

Two kinds of harm

The key distinction (Crawford, Blodgett et al.):

• Allocational harm — a system unfairly allocates resources or opportunities: an LLM résumé-screener that ranks one demographic lower, a chatbot that gives worse advice to some users. This is the classifier-style harm, and the formal criteria from ML fairness (demographic parity, equalized odds) apply.
• Representational harm — the system represents groups in stereotyped or demeaning ways, independent of any decision: completing “the nurse… she” and “the CEO… he,” producing more toxic text for some names or dialects, or erasing a group entirely. There’s no “decision” to audit — the output itself is the harm — and this is where LLMs are distinctively risky.

Where it comes from

Two compounding sources. First, training data: models learn from internet-scale text that encodes real societal bias, and they don’t just mirror it — they can amplify it, sharpening a skew in the data into a stronger one in the output. Second, alignment: RLHF optimizes a human-preference proxy, which can introduce biases — politeness norms, political lean, or sycophancy — that reflect the raters, not truth. Bias isn’t only in the pretraining data; the fine-tuning can add its own.

Measuring it: skew you can count

Representational bias sounds fuzzy, but you can measure it — e.g. across many generations mentioning an occupation, count how the model genders it:

# Representational bias as measurable skew: across the model's mentions of each
# occupation, what share did it gender male vs female? (Illustrative counts.)
outputs = {  # occupation: (male%, female%) in the model's generations
    "nurse":    (12, 88),
    "engineer": (90, 10),
    "teacher":  (40, 60),
    "ceo":      (85, 15),
}
for job, (m, f) in outputs.items():
    skew = (m - f) / (m + f)        # +1 all-male, -1 all-female, 0 balanced
    tag = "skew MALE" if skew > 0.3 else "skew FEMALE" if skew < -0.3 else "~balanced"
    print(f"{job:<9} male {m:>3}% female {f:>3}%  skew={skew:+.2f}  {tag}")

nurse     male  12% female  88%  skew=-0.76  skew FEMALE
engineer  male  90% female  10%  skew=+0.80  skew MALE
teacher   male  40% female  60%  skew=-0.20  ~balanced
ceo       male  85% female  15%  skew=+0.70  skew MALE

The field standardizes this with benchmarks: WinoBias / WinoGender (does coreference flip on gendered pronouns?), BBQ (bias surfacing in question-answering), StereoSet and CrowS-Pairs (does the model prefer the stereotyping sentence?), BOLD (bias in open-ended generation), and embedding tests like WEAT. None is complete; together they turn “the model seems biased” into numbers you can track.

Mitigation is value-laden

There are levers — balance/curate the training data, steer with RLHF / constitutional AI, add system-prompt instructions, and evaluate / red-team for bias before shipping — but none is a clean fix, and they trade off:

In one breath

• Classical fairness audits a classifier’s decisions; LLMs add a generative problem — producing, amplifying, and normalizing stereotypes.
• Allocational harm = unfair allocation of resources/opportunities (classifier-style, uses formal fairness criteria); representational harm = demeaning/stereotyped depiction where the output itself is the harm — LLMs are dominated by the latter.
• Sources compound: training data encodes and gets amplified, and RLHF can introduce biases (politeness, political lean, sycophancy).
• It’s measurable — generation skew (the demo: nurse→female, CEO/engineer→male) and benchmarks (WinoBias, BBQ, StereoSet, CrowS-Pairs, BOLD, WEAT).
• Mitigation (data, RLHF/constitutional, prompting, eval) is value-laden: under- vs over-correction, and fairness is contested — define the harm for your use case and measure, don’t chase a single “unbiased” switch.

Quick check

Next

This extends ML fairness into generation; the RLHF that can both reduce and introduce bias is alignment, and the proxy-gaming behind sycophancy is reward hacking.