An agent's self-reported confidence is poorly calibrated, yet it must decide whether to answer, ask a clarifying question, defer to a human, or take an action. How would you make that decision under uncertainty?

How to think about it

I would not threshold the agent’s self-reported confidence. I would estimate outcome probabilities from task-specific evidence and historical results, then choose among answering, clarifying, escalating, or acting by expected loss, with hard safety and authorization constraints for consequential actions.

The key is to separate what the agent says it believes from what the system has learned happens when it says that.

Why confidence alone is not the decision

Calibration means that a predicted probability matches the observed frequency. Among cases assigned 0.8, a calibrated system should be correct about 80 percent of the time. An agent saying “I am 80 percent confident” does not establish that. Language models are often good at producing a convincing number and bad at making that number mean 80 percent.

That number may still be useful for ranking. Cases scored 0.8 may genuinely be easier than cases scored 0.4. But ranking is not calibration. A score can order cases correctly while being systematically too optimistic.

I would therefore build a decision-specific risk estimate. For an answer, estimate the probability that the answer is materially wrong. For an action, estimate the probability that the action is unauthorized, based on the wrong state, or fails to achieve its intended result. For clarification, estimate how likely the missing information is to change the decision and how likely the user is to provide it. Those are different probabilities. One confidence scalar should not control all four routes.

The decision then becomes:

choose the action with the lowest expected loss

Expected loss is the average cost of an action across the possible states of the world. In shorthand, I would use argmin E[L(action, outcome) | evidence], then apply hard rules that cannot be traded away for a better average.

The evidence should include more than the model’s self-report:

• retrieved documents and whether they actually support the claim;
• agreement between independent checks or a verifier;
• tool results and their freshness;
• ambiguity in the user’s request;
• policy and authorization checks;
• how similar cases performed historically;
• whether the action is reversible.

A useful production policy has four stages: generate candidate responses and actions, estimate their risks, score their costs, and enforce safety gates before execution.

A concrete example: the duplicate $180 charge

Imagine a support agent receives this request:

“Please refund the duplicate $180 charge from Acme.”

The agent’s raw confidence is 0.86. An audit of previous cases shows that predictions between 0.80 and 0.90 were correct only 62 percent of the time. So 0.86 is not used as a probability. It is merely one feature.

The agent checks the ledger and finds two Acme charges on the same day. One is settled. The other is still pending. The refund policy allows refunds for settled duplicate charges but not for pending authorizations. A separate checker confirms the transaction IDs, but the user did not specify which charge they mean.

After these checks, the system estimates a 74 percent probability that the requested charge is both the duplicate and eligible for a refund. That estimate should come from a held-out evaluation set or a task-specific risk model, not from the language model’s prose.

Suppose the support policy assigns these illustrative costs:

Choice	Expected loss at probability 0.74
Refund immediately	(1 - 0.74) x $180 = $46.80
Ask for the transaction ID	$10 + (0.10 x $12) = $11.20
Send to a human	$12
Give a bare answer without resolving the request	$25

The clarification costs an estimated $10 in delay and user friction. It resolves the ambiguity in 90 percent of cases. The remaining 10 percent goes to a human, whose expected handling cost is $12. These values are policy inputs, not laws of nature. A bank may price a mistaken refund very differently.

The rational choice here is to ask one specific question:

“I found a settled charge and a pending authorization for $180. Which transaction ID should I review for the refund?”

That is better than saying “I am 74 percent confident” and hoping a threshold does the thinking.

Now suppose the user identifies the settled transaction, the ledger confirms it is a duplicate, and the authorization check passes. The probability of eligible duplication rises to 0.96.

The expected loss of acting becomes (1 - 0.96) x $180 = $7.20, lower than the $11.20 clarification path and the $12 human-review path. The agent can issue the refund, provided it has permission and the refund operation is safe to retry.

That last condition matters. A tool timeout does not prove that a refund failed. The system should check the transaction state before retrying. Otherwise, the agent can make one correct decision and still create a duplicate refund during execution.

Warning: a high confidence score is not permission to act. Authorization, policy, and precondition checks are hard gates. A system should not refund a charge merely because the expected dollar loss looks acceptable.

How I would build it in production

First, log every decision with the raw confidence, evidence features, selected route, policy checks, tool outcome, and eventual ground truth. Ground truth might be a reviewer’s decision, a corrected answer, a successful transaction, or a user dispute. Without outcome labels, calibration becomes a storytelling exercise.

Second, calibrate on time-split, task-specific data. A calibration method such as isotonic regression or logistic scaling can map a raw score to an observed probability. The data must represent the current traffic, because a model calibrated on last year’s password-reset requests may be badly calibrated on today’s account-recovery attacks.

Calibration should also be measured separately for answering, clarification, human escalation, and actions. A single overall accuracy number can hide a dangerous pocket. The agent might be reliable on product questions and reckless with account changes.

Third, choose thresholds from the cost model, not from a fashionable value such as 0.8. For a low-stakes factual answer, a small chance of error may be acceptable if the answer cites its source and clearly states uncertainty. For an irreversible account deletion, even a 99 percent estimate may be insufficient without a confirmation step.

Clarification is appropriate when one missing fact has high value of information. That means learning the fact is likely to change the decision enough to justify the question’s cost. It is not appropriate to ask vague questions merely to make the agent feel safer. “Can you provide more details?” transfers the uncertainty to the user and usually produces another vague turn.

Human deferral is appropriate when the remaining uncertainty is consequential, the case falls outside policy, or the system detects adversarial or conflicting evidence. It is not a magic oracle. Human queues have latency, cost, and their own error rates, so those belong in the loss model too.

The senior-level nuance

Expected loss is not enough when the loss distribution has a dangerous tail. A one percent chance of exposing a patient’s medical record is not necessarily acceptable because the average cost looks small. Hard constraints should reject actions involving missing authorization, privacy violations, security-sensitive tool use, or irreversible changes.

There is also a difference between uncertainty about the facts and uncertainty about the model itself. If the system has never seen a new kind of invoice, its probability estimate may be unreliable even after calibration. Under distribution shift, widen the safety margin, require stronger evidence, or defer.

The common failure mode appears first in operations, not in a calibration plot: the agent reports high confidence, users correct it repeatedly, and high-confidence actions generate reversals or manual cleanup. The fix is not simply lowering the confidence threshold. Inspect which evidence was missing, add an outcome label, recalibrate that route, and tighten the action gate.

What they’ll ask next

“What if you have very little labeled data?”
Start with conservative rules for high-impact actions, run the agent in shadow mode, and collect labels from reviewers and tool outcomes. Use confidence intervals around small-sample estimates rather than pretending a handful of examples are precise. Low-stakes answers can operate while the system gathers evidence; irreversible actions should wait.

“Would you use the language model’s confidence at all?”
Yes, as a feature or ranking signal. I might combine it with verifier agreement, retrieval quality, ambiguity signals, and tool-state checks. I would not treat it as a probability until it has been validated against outcomes.

“How do you prevent prompt injection from forcing an action?”
Treat retrieved pages, emails, and tool content as untrusted data, not instructions. Require authenticated user intent, explicit authorization, policy checks, and action-specific preconditions outside the model’s generated text. A prompt can influence a proposal; it must not bypass the gate.

One line to say in the room

“I would replace the agent’s confidence threshold with calibrated, action-specific risk estimates, choose the lowest expected-loss route, and let hard safety and authorization constraints override the arithmetic.”