Your offline benchmark score improves after a model change, but production resolution rate falls. How would you investigate benchmark leakage, distribution shift, evaluator bias, and slice-level regressions before deciding whether to roll back?
I would freeze the rollout, verify the production metric and instrumentation, then compare old and new models on a clean recent sample while auditing leakage, distribution shift, evaluator bias, and slice regressions. I would roll back immediately for material safety or customer harm, but otherwise use the evidence to decide between rollback, targeted gating, or a corrected evaluation.
How to think about it
I would freeze the rollout, verify the production metric and instrumentation, then compare old and new models on a clean recent sample while auditing leakage, distribution shift, evaluator bias, and slice regressions. I would roll back immediately for material safety or customer harm; otherwise I would use the evidence to choose between rollback, targeted gating, or a corrected evaluation.
Why the two numbers can disagree
An offline benchmark and a production resolution rate often measure different things.
An offline benchmark is a fixed collection of cases scored under controlled conditions. Production is a moving stream of users, tools, policies, latency, retries, and failures. An agent can become better at the benchmark’s wording while becoming worse at handling a real customer whose identity service times out halfway through an account-recovery flow.
The key question is not “which number is right?” It is “which part of the system changed, and which measurement is sensitive to it?”
I would first check for a simpler explanation: a routing change, broken instrumentation, changed resolution definition, different traffic mix, tool-version change, or immature labels. For example, comparing conversations closed today with conversations that had seven days to resolve can manufacture a regression.
I would define the production metric precisely before analysing it. The denominator might be eligible conversations opened during a period. The numerator might be conversations resolved within 72 hours and not reopened within seven days. The exact rule depends on the product, but it must be identical for both model versions. “Resolved” cannot quietly mean “the agent said it was done.”
The investigation I would run
1. Protect customers and establish a fair comparison
I would stop increasing traffic to the new model and preserve the old model as a control. If the new model can cause unsafe actions, unauthorized refunds, or irreversible tool calls, I would roll it back first. Investigation is not a reason to leave a known harmful system live.
Then I would compare old and new models on the same recent production cases. A replay can use recorded tool responses for analysis, but it cannot fully reproduce user behaviour or mutable systems. A user who receives a confusing answer may abandon the conversation; a replay will not capture that. So I would pair replay with a small, randomized canary when it is safe.
I would also check whether production resolution has delayed labels. A model that resolves cases faster can look worse for several days if the metric counts only mature outcomes.
2. Audit benchmark leakage
Benchmark leakage means information from the evaluation set, its answers, or a near-duplicate has reached the model or the prompt used to evaluate it.
I would inspect the benchmark’s provenance and compare it with training data, fine-tuning data, prompt templates, demonstrations, retrieval indexes, and tool traces. For an agent, leakage is broader than memorizing the final answer. A test case might include a familiar ticket ID, a gold tool response, or wording that tells the model which action the evaluator expects.
I would run three checks:
- Exact and semantic deduplication against available training and tuning data.
- A fresh, private, time-separated holdout written after the model’s data cutoff.
- Paraphrased and adversarial versions where the task stays the same but the wording, names, order of details, and tool responses change.
If the new model’s gain disappears on the private holdout or after paraphrasing, I would treat the benchmark improvement as suspect. I would not necessarily call it intentional cheating. Contamination can enter through a public support corpus or a vendor model’s pretraining. The practical result is the same: that benchmark no longer estimates generalization.
3. Measure distribution shift
Distribution shift means that production cases or operating conditions differ from the benchmark. I would compare the two populations across intent, language, customer tier, channel, conversation length, tool availability, tool latency, policy version, and failure status.
A useful first pass is a model-by-slice table, not one average. I would include sample counts and confidence intervals, because a six-point change on 40 cases is not the same evidence as a six-point change on 40,000.
I would look for both covariate shift and task shift. Covariate shift means the inputs changed: production has more Spanish conversations, longer histories, or more tool timeouts. Task shift means the relationship between an input and a successful action changed: a policy update may have made yesterday’s correct refund action wrong today.
I would also check whether the model itself changes the traffic it sees. If the new agent escalates easy cases early, the remaining queue becomes harder. If it closes conversations prematurely, the measured denominator may change. The model is then part of the data-generating process, not merely a passenger in it.
4. Test evaluator bias
Evaluator bias means the scoring process consistently favours one style or behaviour rather than the outcome that matters.
An automated language-model judge may reward a polished explanation, agreement with a reference answer, or a particular tool-call format. That can inflate the new model if it is more verbose or has learned the benchmark’s preferred phrasing. It may still fail to complete the actual task.
I would separate outcome checks from language-quality checks. If the agent needed to update an address, inspect the resulting system state. If it needed to send a confirmation, inspect the event log. Do not ask a judge to infer a database mutation that the database can verify directly.
For subjective dimensions such as helpfulness, I would have blinded human reviewers score old and new outputs in randomized order using a rubric tied to customer outcomes and policy. I would measure agreement between reviewers and adjudicate disagreements. A judge that strongly prefers the new model but disagrees with customer outcomes is evidence about judge bias, not evidence that the model works better.
A concrete example
Consider Atlas, a hypothetical support agent. Its offline set contains 10,000 conversations: 80 percent routine FAQ cases and 20 percent account-recovery cases.
| Evaluation | Old model | New model |
|---|---|---|
| Offline benchmark | 69.0% | 77.2% |
| Production resolution | 56.4% | 50.2% |
The offline result looks excellent. It comes from these slice rates:
- FAQ: old 75 percent, new 82 percent.
- Account recovery: old 45 percent, new 58 percent.
But production contains 40 percent FAQ and 60 percent account recovery. On a recent replay, the recovery result changes:
- Healthy identity tool: old 50 percent, new 55 percent.
- Identity tool timeout: old 35 percent, new 10 percent.
The new model has learned to continue with a confident answer when the identity tool fails. The benchmark always returns successful tool responses, so it never exercises that path. Production does. The benchmark gain is real for routine cases, but irrelevant to the failure-heavy population that now dominates traffic.
I would verify this with tool logs, not just transcripts. I would check timeout frequency, retries, fallback behaviour, and whether the new model makes a different number of tool calls. Then I would run old and new models on the same timeout traces, review the high-risk failures, and test a guarded policy: when identity verification is unavailable, stop and escalate rather than improvise.
The final decision might be a targeted rollback of the new model for recovery and timeout slices, while keeping it for routine FAQ traffic. If the failures involve privacy or unauthorized account changes, I would use a full rollback until the guardrail is proven.
The senior nuance
A production regression does not automatically mean the model should be discarded. It may expose a benchmark that was too easy, an evaluator that measured style, or a deployment condition the new model was never trained for. Conversely, a clean leakage audit does not rescue a model that harms real users.
I would make the decision using customer impact, not statistical drama. A broad one-point gain cannot compensate for a severe regression in a small but safety-critical slice. I would also account for uncertainty, delayed outcomes, cost, latency, escalation burden, and whether a fix can be safely isolated.
The most useful experiment is usually a paired, randomized comparison on recent traffic with stable instrumentation, plus predeclared slices. That tells me whether the model caused the change. The benchmark audit tells me whether I trusted the wrong evidence in the first place.
What they’ll ask next
How do you distinguish distribution shift from a model regression?
Run both model versions on the same recent cases and the same recorded tool conditions. If the new model loses on identical cases, it is a model or integration regression. If both perform similarly but production worsened because the case mix changed, it is primarily shift. Often both are present.
What if production labels arrive weeks later?
Use mature cohorts for the main metric, report provisional metrics separately, and sample recent cases for blinded human review. Proxy signals such as repeat contact, escalation, tool success, and reopen rate can guide triage, but they are not substitutes for the final outcome.
How do you avoid finding a fake regression by checking hundreds of slices?
Define important slices before looking at results, require minimum sample sizes, show confidence intervals, and replicate surprising findings on a fresh time window. Treat a slice as actionable when its effect is large, plausible, repeated, and costly—not merely because one p-value happened to look exciting.
One line to say in the room
“I would not let a higher offline score overrule production evidence: I’d freeze the rollout, audit contamination and evaluator validity, compare old and new on the same recent cases, and make the rollback decision from outcome-weighted slice results.”