What biases affect LLM-as-a-judge evaluations, and how do you mitigate position bias?
LLM-as-a-judge systems can be biased by answer order, verbosity, style or self-preference, and rubric or reference anchoring. Mitigate position bias by blinding and swapping candidate order, treating order-sensitive results as inconclusive, and validating aggregate scores against human-labeled examples.
How to think about it
An LLM-as-a-judge is a language model used to score or compare other model outputs. It is useful, but not neutral: answer order, length, writing style, the rubric, and the judge’s own preferences can all affect its decision. I mitigate position bias by hiding provenance, evaluating both candidate orders, treating order-sensitive results as inconclusive, using an explicit rubric, and calibrating the whole setup against human-labeled examples.
Why these biases appear
A judge does not calculate a timeless measure of answer quality. It generates a likely judgment from the prompt and the text in front of it. Change the prompt context, and you can change the judgment even when the candidate answers are identical.
Position bias is the tendency to prefer the answer in a particular position. Some judges favor the first answer. Others favor the second, perhaps because of recency or because the final option feels more salient. The important point is not which position wins. The important point is that the winner can change when the positions change.
Verbosity bias is the tendency to mistake length for quality. A 500-word answer can look more thoughtful than a precise 120-word answer because it contains more explanations, qualifications, and familiar signals of effort. But extra words may merely repeat the claim or bury an error. The reverse problem also exists: a short answer can be correct but omit the reasoning the task requires.
Style or self-preference bias is the tendency to reward wording that resembles the judge’s preferred style, or even outputs produced by the same model family. A judge may prefer polished, hedged prose over a terse but accurate answer. This is especially dangerous when comparing models with different writing styles. The judge can end up measuring “sounds like me” rather than “solves the user’s problem.”
There are other biases worth naming in an interview:
- Rubric anchoring: the judge follows a vague or leading rubric instead of the task. “Which answer is more helpful?” leaves a great deal of room for taste.
- Reference-answer anchoring: when shown a reference, the judge may reward lexical similarity even when a different answer is equally correct.
- Position-independent judge error: the model may confidently accept a false claim because it lacks domain knowledge. Swapping order cannot fix that.
- Stochastic variance: repeated calls can disagree because generation is probabilistic, even with identical inputs. Setting sampling randomness low can reduce this noise, but it does not remove systematic preferences.
That distinction matters. Position bias is an order effect. It is not the same thing as a judge simply being wrong.
A concrete position-bias test
Imagine an evaluation for a customer-support model. The policy says refunds go to the original payment method within 30 days, and downloaded digital goods are not refundable.
Candidate A says:
“Refunds are sent to the original payment method within 30 days. Downloaded digital goods are not eligible for a refund.”
Candidate B is warmer and longer, but incorrectly says refunds are available within 60 days.
Use a ten-point rubric:
| Criterion | Points |
|---|---|
| Policy accuracy | 5 |
| Compliance with the requested action | 3 |
| Clarity and relevance | 2 |
Now compare the same two answers twice:
| Run | Visible order | Judge result |
|---|---|---|
| 1 | A, then B | A: 8, B: 7 |
| 2 | B, then A | B: 8, A: 7 |
The winner changed even though neither answer changed. That is a direct diagnostic for position sensitivity. The numbers are illustrative, but the test is real and easy to run.
At scale, suppose 200 candidate pairs are evaluated in both orders and 36 pairs produce a different winner after the swap. The swap-disagreement rate is 36 / 200 = 18%. An aggregate win rate may still look impressive, but an 18% disagreement rate means many individual decisions are not reliable enough to use as labels or deployment gates.
The conservative aggregation rule is:
def stable_winner(first_order_winner, reversed_order_winner):
if first_order_winner == reversed_order_winner:
return first_order_winner
return "inconclusive"The two inputs must refer to the original candidate identities, not merely “the first slot” and “the second slot.” If A wins both orders, count A as a stable win. If B wins both, count B. If each wins once, report an inconclusive result, a tie, or send it to human review.
How I mitigate it in production
First, I randomize and blind the comparison. The judge sees neutral labels such as A and B, not “new model,” “baseline,” or a model name. Candidate order is randomized so that a source, model version, or dataset shard is not always placed in the same position. This prevents a systematic source-position correlation, but randomizing only once is not enough for a dependable pair-level decision.
Second, I perform a paired order swap. For every pair, I run the exact same judging prompt with the candidates exchanged. I keep the rubric, user request, formatting instructions, and candidate text unchanged. I then record both the final winner and each criterion score.
Averaging the two scores can reduce an order effect in an overall mean, but it can also hide instability. Therefore I report both:
- the order-adjusted average score;
- the fraction of pairs whose winner changes after swapping.
That second number answers the operational question: “Can I trust this particular comparison?”
Third, I use a rubric with observable criteria. Instead of asking “Which answer is better?”, I might ask the judge to score factual accuracy, instruction following, completeness, and relevance separately, then make a final comparison. A rubric does not make the judge objective, but it narrows the space of arbitrary preferences. For factual or executable tasks, I prefer deterministic checks where possible: tests for code, exact constraints for structured output, or verified reference facts for a narrow question.
Fourth, I aggregate cautiously. Multiple samples from one judge can estimate variability. Multiple judge models can reduce a model-specific style preference. Neither creates independence automatically: models trained on similar data can share the same blind spot. I would not call a majority vote “ground truth” merely because five judges agreed.
Finally, I calibrate against humans. I create a fixed set of examples with human labels, including clear wins, genuine ties, borderline cases, different answer lengths, and different writing styles. I measure judge-human agreement, swap disagreement, and error rates by task type and language. I inspect disagreements rather than looking only at one headline score. If the judge consistently rewards long answers or misses a particular factual error, I revise the rubric, change the judge, or add a deterministic check.
For a high-stakes decision, an LLM judge should usually be a filter or assistant, not the only authority. A fluent judge can confidently approve a harmful medical recommendation or an insecure code change. Position debiasing fixes one failure mode; it does not give the model domain expertise.
The senior-level trade-off
Order swapping costs roughly twice the judge calls. Ten thousand pairwise comparisons become twenty thousand calls before retries, repeated samples, or judge ensembles. That affects latency and token cost.
Counting only consistent wins is also conservative. It turns some real but close wins into ties, reducing statistical power. That is often the right trade when labels will train another model or decide a launch. For dashboarding, I might retain the average score while prominently reporting the inconsistency rate. For automatic promotion, I would require a stable win or human review.
I also would not force candidates to have the same length just to remove verbosity bias. Length can be part of the task: a legal explanation may need detail, while a command-line answer may need one command and one warning. The rubric should distinguish useful completeness from repetition. Removing a genuine quality signal is not debiasing; it is changing the task.
What they’ll ask next
Does setting temperature to zero solve position bias?
No. It can reduce random variation between repeated calls, but a deterministic judge can still prefer whichever candidate appears first. Position swapping is needed to measure and control that systematic effect.
Why not randomize the order once and trust the aggregate win rate?
Randomization helps cancel a position preference across a large sample, but one evaluation still gives an unstable answer for a particular pair. A candidate may win because it was placed first. Paired swaps expose that fact and let you report inconclusive comparisons instead of manufacturing certainty.
How do you evaluate the judge itself?
I use a human-labeled calibration set and measure agreement, swap-disagreement rate, false wins on known examples, and performance by task type and subgroup. I also inspect the judge’s reasons and criterion scores, but I do not treat persuasive explanations as evidence that the verdict is correct.
Say this in the interview
“LLM judges have position, verbosity, and style or self-preference biases; for position bias I blind the candidates, swap their order, count only consistent wins or flag disagreements, and validate the rubric and aggregate results against human labels.”